Hacker News new | past | comments | ask | show | jobs | submit login
Google is Breaking the Internet (jeremypalmer.com)
503 points by kposehn on May 18, 2014 | hide | past | favorite | 297 comments



I own http://ihackernews.com which reformats HackerNews for mobile phones.

I had a DMCA takedown notice sent to me on behalf of a website owner who didn't want me linking to their site. My hosting provider gave me 24 hours to remove the link or else they'd cancel my account.

The owner of the link claimed that their Google rankings were dropping because my site, iHackernews, linked to their site. With this, they were able to force me to remove it via a DMCA takedown notice.


Why didn't you challenge the takedown notice as being fraudulent? As far as I know, DMCA does not prohibit linking to web pages; it's only about hosting someone else's copyrighted content on your site (or maybe posting links to pirated content).


Challenging a DMCA takedown requires you to accept that your whole site will be taken down for ten days while you wait to have a right to challenge the takedown. Businesses or projects that have to keep functioning have to comply with any takedown regardless of its validity because they can't just go out of business for ten days in order to challenge it.


Wouldn't temporarily removing the disputed item pending the resolution of the challenge prevent your entire site from being taken down?


I just don't have time to deal with it. Removing it was the quickest way possible for me to resolve the issue and not have my account closed.


and, if they're abusing the DMCA like others have said, you're giving in to FUD, which is exactly what they want.


You don't even need to formally challenge it. Any DMCA takedown request requesting that a link be removed is malformed, becuase it does not identify a copyrighted work that is being infringed. You can safely ignore such a request, or bounce it back requesting that the sender specify what copyrighted work is in question.


Yes, but the person receiving the DMCA request was the hosting service, which had no intention of being sane.

DMCA takedown notices cannot be ignored.


Could you please tell us your hosting provider and the website that sent the DMCA notice? I'd like to avoid both of them, if I can.



Ohh the irony of them being a "Link removing service". I wonder how it would have affected the credibility & guarantee of their service had you told them no.


I had a DMCA takedown notice sent to me on behalf of a website owner who didn't want me linking to their site.

How does that even work? How does the DMCA apply such that I can invoke it to tell you to stop linking to my site?

I'm guessing it doesn't, but if you're at the mercy of a hosting service that doesn't actually understand the DMCA you're in a jam.


The way the DMCA works ISPs or any other service hosting third-party content must comply with every takedown notice immediately, regardless of it's legitimacy, if they want to continue to enjoy the protections of the DMCA. (meaning, unless they want to accept liability for the copyright of every piece of content on their servers). So yes, it absolutely does work that way: you can send a takedown notice to anybody for any reason and they must comply with it.

The logic is that DMCA notices are sent under penalty of perjury, and people aren't going to risk perjuring themselves just to improve their SEO or take down a blog post they don't like. Unfortunately, the courts seem willing to accept excuses like "the automated process i use to send out DMCA takedown notices got a little overzealous" as legitimate, so DMCA takedowns are now pretty much just a free-for-all for anybody who wants anything taken down.


> The way the DMCA works ISPs or any other service hosting third-party content must comply with every takedown notice immediately, regardless of it's legitimacy, if they want to continue to enjoy the protections of the DMCA.

AFAICT from actually reading the act, they only have to comply with notices meeting the facial requirements, and failing to do so only exposes them to liability for the content covered by the particular notice for which they failed, and only where they would have had liability without the DMCA safe harbor provisions. I'd like to see any analysis supporting the characterization you make here.


Someone needs to send a couple of millions DMCA takedown requests from random sources to random targets, to make this problem visible.


Hurting innocents in the pursuit of political reform isn't the way to go.


The hurt was imposed by the law.

Utilizing the law to its letter and to impose the fundamental flaw in it is simply making that hurt visible.


The sending itself wouldn't be hurting anyone. The responding parties would have to make a decision to hurt innocents or not, and maybe actually forced to do some due diligence, that is expected of them. And maybe voice out that current requirements make this hard or impossible for them.


Several places in the world have shown that to be effective. And that would be real physical harm.


When has terrorism been an effective way to invoke change in a democratic society?



Colonists objected to the Tea Act because they believed that it violated their rights as Englishmen to "No taxation without representation," that is, be taxed only by their own elected representatives and not by a British parliament in which they were not represented.

Again, and in hopes of a more substantive reply: when has hurting innocents been an effective way to get what you want in a society with political representation for its citizens?


Ok, how about not fully random sources and targets, but ones that somehow involved in DMCA lobbying and enforcement. That would remove the "innocent" from equation.


The IRA has members in parliaments now. Negotiations with the Taliban have changed their method from arms to words. I could probably find a few more by spending a little time researching.


It doesn't apply. A DMCA request must identify a copyrighted work that is being infringed. If your host is forwarding on these takedown requests, they're not doing their job properly.


One of the worst decision in Google search history: to base ranking on DMCA's received...

It's bad enough that they have automated the process to allow companies to delete millions of links from Google every year, but to downrank sites is a bit overboard.


I didn't know they did that, I wonder how strong a signal it is. If you search for a review or something of a recent TV show (my favourite is eg game of thrones S4E5) and forget to mention you want reviews, you find lots of sites that must be getting DMCAs. Also Youtube must get a lot of them, probably one of the top served sites at a guess?


Do you have evidence that happens? That sounds very strange.


What a terrible hosting provider you have. Who are they?


Not sure when the post was made, but ronnier later replied that it's http://www.softsyshosting.com.

I definitely agree though - they sound like a very poor host. One of my websites received an invalid DMCA via my host, and my host handled it well (i.e. listening to my side of the story, before deciding not to pursue the 'complaint').

Any host that handles DMCAs so poorly (i.e. threatening their customers without knowing the facts) doesn't deserve to be in business IMO.


Interesting. Good to know how to deal with this problem should it arise.

In your case obviously it was unjust though.


I talked about this phenomenon recently on This Week in Google with Leo Laporte and Gina Trapani: http://twit.tv/show/this-week-in-google/248 Skip to 4:15 in to listen to the discussion.

Note that there are two different things to keep in mind when someone writes in and says "Hey, can you remove this link from your site?"

Situation #1 is by far the most common. If a site gets dinged for linkspam and works to clean up their links, a lot of them send out a bunch of link removal requests on their own prerogative.

Situation #2 is when Google actually sends a notice to a site for spamming links and gives a concrete link that we believe is part of the problem. For example, we might say "we believe site-a.com has a problem with spam or inorganic links. An example link is site-b.com/spammy-link.html."

The vast majority of the link removal requests that a typical site gets are for the first type, where a site got tagged for spamming links and now it's trying hard to clean up any links that could be considered spammy.

If you read the original post closely, it's clear that this is a site asking for a link to be removed--the quoted email isn't from Google.


Matt, this does not address whatsoever the point being brought up. And in case you missed it, I will try to cite it as clearly as possible here:

This is a case where google wants people to mold the internet according to what their current incarnation of their algorithm says the way it should be.

Rather than google trying to mold its algorithm to understand the internet, natural links vs. spams, and their respective authority and value.

Google here is saying that all links from this site are suspect, even when the link in question is 100% valid, quality and relevant.

Instead of going back to the drawing board and trying to fix the algorithm, google has discovered that since they are so big now, it is easier to play the "benevolent" dictator and dictate to the whole internet, how and what they should be doing.

And to mete out knuckle raps when they get out of line.

Do you see the point now?

The model (google's algo) should fit reality. Rather than forcing reality to fit the model.

I believe this is what the author is trying to communicate by saying "google is breaking the internet" - what he means is that the internet is becoming a 'google' version of itself, rather than what it would naturally be.


> Google here is saying that all links from this site are suspect, even when the link in question is 100% valid, quality and relevant.

I read the article and didn't see any mention of what site the link was on or even what site it was linking to. It's impossible to say if it was legitimately spammy or not, but either way Google never asked him to do anything. Someone who didn't know about the Link Disavow Tool decided on their own to email him about taking the link down.

You would have a good point prior to the Link Disavow Tool, but it has solved this (very tricky) problem.


Someone who didn't know about the Link Disavow Tool decided on their own to email him about taking the link down.

FTFA:

"In short, the email was a request to remove links from our site to their site. We linked to this company on our own accord, with no prior solicitation, because we felt it would be useful to our site visitors, which is generally why people link to things on the Internet."

"Apparently Google convinced them, via their Webmaster Tools portal, that the link looked 'unnatural', and that they should use the Link Disavow Tool to discredit the link. Furthermore, they thought it necessary to contact us to manually remove the link, which is something I’m not going to do (out of principle)."


... Exactly. Someone took it upon themselves to hassle the OP to remove a link. Google has made a tool to allow you to have links that you don't vouch for to be not counted, but the webmaster in question decided to try and do it "manually".

It used to be you would get sketchy emails asking you to link XYZ and now a few years in the future the sketchy emails are asking you to unlink XYZ.


I think you missed the point of the disavow tool. If Google hit you with a penalty, they still expect you to put in a good faith effort to remove the link naturally first.

https://support.google.com/webmasters/answer/2648487?hl=en


the doc says it's your responsibility to remove spammy back links that you (or a SEO you've hired) have introduced to improve your ranking.

it cannot possibly be your responsibility to remove links that other unrelated entities have towards you! (unless it's an obvious link scheme)


I think Matt and Google are getting an unfair rap on this. Google provided a search engine that worked better than any other so people started using it. People are still using it because it's still better than the others. It's not a monopoly. You can use another search engine if you want to.

The site that emailed to get a link taken down employed an SEO agency to improve their rankings and they created a bunch of spammy links to try to manipulate Google into ranking that site higher than others that probably deserve to rank higher due to their content. If Google did nothing about it we would get some crappy search results.

Also, if you read what Matt Cutts is saying, it implies that if they only had a few unnatural links then they wouldn't have received a webmaster tools warning telling them they had spammy links. They got the warning because their link profile suggested that it was obvious they had been trying to manipulate rankings, not because they had one or two iffy ones.

It's likely that the site then blanket emailed anything and everything that they weren't sure about to get the links taken down. Is it Google's fault that this is how the site decided to deal with the warning? Should Google just make it really easy for them by saying, "OK you disavowed those crappy links, you're off the hook now"? If the consequences are so weak and short-lived, where is the deterrent effect? No, Google doesn't own the internet but they own Google and the site is just trying to provide users with the most relevant results to their searches.

Re, the nofollow aspect. Google didn't "force" sites into using the nofollow attribute by scaring them into thinking that their rankings would drop. Nofollow was in place before there was any notion that who you linked to could harm you. Sites like Wikipedia and others were only too happy to add nofollow to links because it put spammers off from crapifying their sites.


No, they aren't. Google is using FUD to scare sites into requesting links be removed that they never built on legitimate sites. No one can possibly guess which links an individual Googler will think "might" be paid. I assure you they guess wrong most of the time.

As Google penalizes the best writers and blogs, one of two things happens:

1) Entire sites are being nofollowed - removing links to deserving sites.

2) The site refused to nofollow, so they stay penalized with PR0 no matter how good the site may be or how high the PR once was.

The end result will be that even more black hat spammy MFA sites will rank. Sites that deserved to be written about and linked to will drop and be invisible.

It is high time people stop defending Google when they are destroying small businesses and costing much needed jobs. Now Cutts has declared war on bloggers who often make a living writing.

Google wants to destroy any site that enables advertisers and businesses to connect with sites where their target audience already is by forcing publishers to decide whether they want to risk losing organic traffic and being branded with a scarlet PR0 or to nofollow everything they ever write.

Nofollow = NOT TRUSTED. We do not invest time writing about anything we do not trust. (On the VERY rare occasion we do we do not link to them.) We do not publish content that does not fit our audience.

Every link is a potential risk. Nofollowing links may damage the sites we link to so what is a writer who already cares about what they write about supposed to do?

Google has a monopoly on search because the wealthy elite who own the media handed it to them. People use it because in many cases they do not even know there is an alternative. Many use whatever is installed on their computers without even having any idea what it even is.

People who are trying to clean up unnatural links penalties are not SEO experts. They rely on tools to tell them what links to remove. These tools are not very accurate. I've had them flag DMOZ, Best of the Web, and Business2Community as "toxic". Finally many remove and/or disavow every link in desperation.

How does all that disavowing affect search results and individual sites? How many verify that requests to remove organically given links are from the actual site and not their unethical competitor. (Hint: almost none.)

Many businesses fail waiting for recovery. Others get penalties lifted, but having removed links they no longer get any traffic.

"Google didn't "force" sites into using the nofollow attribute by scaring them into thinking that their rankings would drop." YES, they most certainly DID!


Lots of misinformation in this post. It is almost hard to pick a spot to start.

>Google provided a search engine that worked better than any other so people started using it.<

While at the beginning this may have been true, over time that search engine has grown more parasitic in terms of things like scrape-n-displace knowledge graph results, AdWords ads on branded keyword terms where ads with junk bundleware rank above official sites, etc. http://blog.brandverity.com/2738/software-bundlers-target-br...

>People are still using it because it's still better than the others.<

Care to explain why Google is spending over a billion Dollars a year buying default search placement in other browsers like iOS & Mozilla Firefox? Any thoughts on the Flash security updates which hit other browsers and bundle Chrome's web browser in with it? Or how about the Android contracts with default search placement (and other forms of) bundling baked into them?

Google is spending well over a billion Dollars a year on the thesis that your thesis is wrong.

>It's not a monopoly.<

The hell it's not. At least if we use any of the standard definitions.

>You can use another search engine if you want to.<

And while an informed individual may choose to, the majority of people are driven by default settings which are purchased, as per the above.

>The site that emailed to get a link taken down employed an SEO agency to improve their rankings and they created a bunch of spammy links to try to manipulate Google into ranking that site higher than others that probably deserve to rank higher due to their content.<

Are you suggesting there are not false positives, or that competitors do not buy links to torch their competitors? Either such assertion is simply untrue.

>They got the warning because their link profile suggested that it was obvious they had been trying to manipulate rankings, not because they had one or two iffy ones.<

The second they there presumes a competitor didn't do them for it. Only a person ignorant of the field of SEO would presume this to be true in all cases.

>Re, the nofollow aspect. Google didn't "force" sites into using the nofollow attribute by scaring them into thinking that their rankings would drop. Nofollow was in place before there was any notion that who you linked to could harm you. Sites like Wikipedia and others were only too happy to add nofollow to links because it put spammers off from crapifying their sites.<

This is a complete misunderstanding of history, on numerous levels.

Nofollow was introduced as a (ineffective) solution to blog comment spamming. To help aid further/wider adoption, at some point some Googler's even suggested things like pagerank sculpting could be useful, up until some large sites started doing it excessively. They were looking for reasons to justify its widespread use, because Google intended from day one that the tag could then be spread onto paid links & other links they didn't want to count.

And the other level of absolute misunderstanding was that (before Google went on their fearmongering campaign about links) they in the past suggested that you couldn't control who links to you, but you could control who you link to & sites which linked to bad neighborhoods could indeed be penalized for it.

Very rarely is there a comment which is that long & that wrong. Impressive!


A spammy link is a link created on behalf of the target site.

In this case the author says he created the link independently, because he thought it would be useful to his users. That's, by definition, a genuine, organic link.

I wonder if a link flagged by the disavow tool and confirmed by the user weaken the Google juice of the source site.

@MattCutts (or another Googler), is it the case?


"I wonder if a link flagged by the disavow tool and confirmed by the user weaken the Google juice of the source site."

We haven't been using disavowed links as a reason not to trust a source site. So it wouldn't weaken the Google juice of the author's/source site if the company that sent the email had just disavowed the link.


This in no way implies Google won't use the disavow data to not trust source sites in the future.


Good to know, and thanks for the answer.


> In this case the author says he created the link independently, because he thought it would be useful to his users. That's, by definition, a genuine, organic link.

Genuine organic links can still be spammy. The OP seems to be a genuine affiliate marketer/spammer, it's not inconceivable that his links aren't treated with a ton of respect.


Affiliate marketer & spammer are not natural synonyms.

Some entities might try to say it is six of one & a half dozen of the other, http://www.seobook.com/brand-vs-affiliate-vs-spam but those same entities do not view their own affiliate marketing efforts as spam, even when they are pure spam http://www.seobook.com/google-lends-a-helping-hand

Look at the cloaked affiliate YouTube links to iTunes on music videos. And Google has invested in Viglink, RetailMeNot, etc.

In addition to those types of investments, it was Google's ad programs which funded entities like Mahalo & eHow.

There are zero affiliates which have made as much off affiliate marketing as Google has.


That's a genuinely good point :-)


It seems like it would be odd to not incorporate a "spamminess" measure into the strength of the link, so hopefully you'd be losing very little.


"This is a case where google wants people to mold the internet according to what their current incarnation of their algorithm says the way it should be"

You do realize, of course, that Google is part and parcel of the internet, and that much of the way the internet works "in reality" is in response to external perceptions of how googles algorithms work?

Google's early dominance in search was a result of their algorithm's ability to surface relevance. If they had never staked out that Site A linking to Site B means site B has more relevance, then the internet would be an entirely different beast, and SEO would take entirely other forms that do not amount to linkstuffing.

The internet hasn't existed 'naturally' since search engines first came around.


It's not just Google that recommends to avoid link spamming. For example, at http://www.seroundtable.com/bing-links-knowledge-18535.html you can read where Bing said "You should never know in advance a link is coming, or where it’s coming from. If you do, that’s the wrong path."

When SEOs break the quality guidelines of all major search engines (not just Google), then they might need to do some work to clean things up if they get caught.

One of the reasons that Google is such a popular search engine is that we do take spam seriously and we take action to counter spam.


When SEOs break the quality guidelines of all major search engines (not just Google), then they might need to do some work to clean things up if they get caught.

This still seems to be missing the point.

As I understand it, sites that are not doing anything in the realm of SEO can get penalized because of the actions of people outside of their control.

If the wrong sorts of sites (however defined), sites outside my control, are linking to my site, why am I penalized, and why should the burden be placed on me to go fix it?


Frankly a bit worrying that Matt Cutts doesn't seem to get the fairly straightforward point (to the rest of us) being made here. Or he does and the criticism stands. Either way not so great.


Put the pitchfork down; he's just giving non-specific answers in response to non-specific situations. For all we know, the two domains in question are `bestquickdietfatloss.com` and `levitracialisfreedietpillscanadamexicoblog.com`, in which case it would be correct of Google to indicate that the author's link is linkspam.


Even if the sites are not relevant, the existence of a link does not mean the site which the link points at was responsible for creating the link. In some cases (especially with website hacking) those who create the spam will link to innocent third parties to create a smokescreen. Google can & does burn some of these "drive by shooting" sites to a crisp.

Check the people who try to drop profile links & comment spam links on your blog and such. Are there any websites which have as many spammy inbound links as YouTube does? How many people does Google allocate to cleaning up YouTube's spammy link profile?

There's another factor in that Google may shift what is considered reasonable over time. A buddy of mine got an unnatural link warning where the link cited was a link that has been in place LONGER THAN GOOGLE HAS EXISTED.

Now consider that some websites are bought & sold, change ownership, etc. ... does it make sense to penalize today for something which happened 10 or 15 years ago?

Another factor is that just by ranking you will pick up some scraper site inbound links you do not want. Many of those sites in various forms or fashion are monetized via Google ads. Google continues to run their ads on many of those aggregation sites, yet you may get an unnatural link penalty for having links from the same sites.

Good enough for Google, NOT good enough for you TM


Except that I and it seems many commenters here have seen similar things occurring with non-spam sites. Regardless of the specific domains referred to in this post, a general pattern is being talked about - the fundamental criticism is that we are getting a google-shaped web rather than a web-shaped google, due to google's de facto monopoly position. If this wasn't the case, it wouldn't be a talking point here.


"we are getting a google-shaped web rather than a web-shaped google"

Very well put.


We were getting a google-shaped web from the moment SEO started optimising for Google rankings.


Frankly a bit worrying that Matt Cutts doesn't seem to get the fairly straightforward point

Oh he gets it, it's his job to act like he doesn't and to change the subject "...Ummm...Bing does it too..look I made a video"


"If the wrong sorts of sites (however defined), sites outside my control, are linking to my site, why am I penalized, and why should the burden be placed on me to go fix it?"

Well, if a website was link spamming then it seems appropriate that the site should attempt to clean up the spam they made before they can rank well in Google again. Otherwise it's not fair to the other websites that have been trying to rank fairly and would like a level playing ground.

In general, if you weren't trying to create spam links, then it's very unlikely that any of this is an issue for you and the burden isn't on you to fix it. I made a video about the "what if the wrong sorts of sites link to my site" question here: http://www.youtube.com/watch?v=HWJUU-g5U_I


>In general, if you weren't trying to create spam links, then it's very unlikely that any of this is an issue for you and the burden isn't on you to fix it.

So if Bing wants to rank Bing.com higher than Google.com on the keyword "Search Engine", all they need to do is to pay link farms to link to Google?

Like everything this could go both ways. If you penalize sites for having links from link farms, what is stopping your competitors from paying link farms to link to your website? or worse yet, some link farm owner comes up with an ingenious idea to hold other websites hostage until they pay up?

It would be more understanding if Google din't have any other resources on hand to make smart decisions but heck you guys have hands on realtime experience with most traffics via Google Analytics, Google Adsense and even DNS. All these resources and can't even distinguish a link farm from a legitimate site? Doesn't sound believable.


> All these resources and can't even distinguish a link farm from a legitimate site? Doesn't sound believable.

The Google webspam team seems to prefer psychology over technology to solve the problem, especially recently. Nearly everything that's come out of Matt Cutt's mouth in the last 18 months or so has been a scare tactic.

IMO all this does is further encourage the development of "churn and burn" websites from blackhats who have being penalized in their business plan. So why should I risk all the time and effort it takes to generate quality web content when it could all come crashing down because an imperfect and overzealous algorithm thinks it's spam? Or worse, some intern or non-google employee doing a manual review wrongly decides the site violates webmaster guidelines?


>prefer psychology over technology<

yup.

>Nearly everything that's come out of Matt Cutt's mouth in the last 18 months or so has been a scare tactic.<

the bizarre thing is the gap in perceptions internal to Google versus external. https://twitter.com/mattcutts/status/449008706655510528 https://polldaddy.com/poll/7916680/

>IMO all this does is further encourage the development of "churn and burn" websites from blackhats who have being penalized in their business plan.<

Absolutely. churn & burn sites, and then some mixed in parasitic hosting on sites which are already highly trusted.


>So if Bing wants to rank Bing.com higher than Google.com on the keyword "Search Engine", all they need to do is to pay link farms to link to Google?

During this "crackdown campaign" Google has been on, I have sat in on meetings at large clients where they've pulled sites that are supposedly link farms (according to google) and put together an ad buy on behalf of competitors meant to cause these penalties to be levied and WMT messages to be sent.


Sounds like a classic Bait and Switch, albeit a good 10 year long one on Google's behalf.

They needed to provide a useful service to begin with to get the traction (good search algorithms for the web-that-is), but now that they are the incumbent, it is more in their interests to penalise website owners and force them to buy ads.

It's sad, but it's normal monopolist behaviour.


Now you're talking about a malicious attack. If that's what it takes for this to be a real issue then I don't think many people need to be too concerned.


The cost of building a quality website & promoting it can easily hit deep into 6 or 7 figures.

The cost of throwing scummy links at a website is under $100.

And if their first $5, $10, or $20 test doesn't work? Then so what for them. They put it on cron job and throw another $20 at it again and again. http://www.seobook.com/images/sape-sape-sape.gif

The key point here is that there's a 100X or 1000X gap between 4 figures & 6 to 7 figures. The cost of burning things to a crisp is much lower than building them up. Whenever incentive structures have a 1000-fold difference in outcomes the path of least resistance becomes a popular path.


It seems like you are heading down the path where black hat anti-SEO on competitors becomes an increasingly viable business model. For example, company A competes with company B. They are cutt-throat so they pay a blackhat SEO firm to generate obviously spammy links for their competitor, not in an effort to bring up their own ranking, but to tank the ranking of their competitor relative to their own. At the end of the day, you do not know the intent of a link. Discounting a low quality link is reasonable. Penalizing the site it points to is not.


The victim site looks at where these newly created back links are coming from, and then reports them to the Google disavow tool and disavow the lot of them. That is the mechanism for dealing with the intent of those links.

After a few cases like this, a long list of multiple disavowed domains/networks/fingerprints, where are these negative links going to come from? This anti-SEO tactic you outline can only be used a handful of times before it becomes useless.


So now black hat SEO people have a tool where they can buy links and disavow the ones where they get caught?


What Matt Cutts is saying, is that as collectively smart as his employer Google is, they aren't able to always figure out if a link is natural and when it's "paid for" as part of linkspam. If they could, they would obviously put that in their algorithm.

Matt Cutts' job at Google involves him making tons of videos explaining how the Google algorithm works. It isn't omniscient, and the guidelines show how to work within its limitations. If you don't care about optimizing your Google presence (as we don't) then you shouldn't care about Google's guidelines either. But if you do comply with some guidelines, then this will probably help your rank in other search engines which also take similar measures to TRY and detect linkspam.

Google's customers are its users. It tries hard to have the most relevant results (or provide value some other way, or just maintain a brand image so people keep coming back). The advertisers come because the users find them and spend real money. And the users want relevant search results, not spam. That's why Google spends a ton of resources on combating the spam.

How do you propose they magically "know" when a link is natural?


> Google's customers are its users.

No, Google's customers are its advertisers. They're the ones who pay money to Google.

I agree with your general point though, the algorithm isn't perfect and sometimes it goofs. What I think people are having a negative reaction to is Matt Cutts' seemingly obtuse responses, as though he doesn't even understand the issue at hand (Google's actions distorting natural behavior), so he just keeps explaining the anti-spam policies in general terms. It's not a conversation, it's people talking past one another.


The icing on the cake is that Matt Cutts is obviously a very bright and very perceptive person so it's easy to assume that his not understanding is intentional. That would be right out of the PR/Politics "avoiding awkward questions 101" textbook but it's a scummy tactic.


I would bet Matt's IQ is easily far north of 140. He's one of the most perceptive people I've ever met.

Think of some of the stereotypes of engineers. Many can code but struggle to communicate. Matt can code, has his name on many patents, leads a team, makes hundreds or thousands of videos, regularly keynotes at conferences & interacts with thousands of people, was GoogleGuy for years, participates in the comments here, was an expert in some of the past lawsuits against Google ... he has basically had a near infinite number of opportunities to put his foot in his mouth & yet how many times has he ever done it? Almost never, if ever.

About the only times I think he has potentially missed at all was the "breaking their spirits" bit on TWIG, and then two minor bits in a few YouTube videos

http://www.youtube.com/watch?v=muZuX9OaMLo&feature=youtu.be&... "We got less spam and so it looks like people don't like the new algorithms as much."

https://www.youtube.com/watch?v=2jap11IIxfw "the gap between what you can get away with and what google says you can get away with is getting smaller all the time"

---

Who do you know who has as much media exposure as he has had who hasn't repeatedly put their foot in their mouth? For me, that answer is nobody.

---

What makes Matt's performance even more impressive is how scummy some of Google's policies have been, even as he came off smelling like roses. There are endless debates on how reasonable it is for Google to infer intent on links & so on, yet at the same time some of the past Google executive emails have quotes in them like:

http://www.washingtonpost.com/politics/google-faces-new-pres... "As with all of our policies, we do not verify what these sites actually do, only what they claim to do." (for AdWords advertisers)

http://www.theverge.com/2013/1/23/3906310/the-no-hire-paper-... "I would prefer that Omid do it verbally since I don't want to create a paper trail over which we can be sued later? Not sure about this. thanks Eric"


> Google's customers are its users.

No, Google's customers are primarily its advertisers. Optimizing advertising revenue has some overlap with pleasing users, as you note, but if a decision needed to be made between pleasing users and increasing revenue, there's no doubt about which way it would go.


They would choose pleasing the users.

I suppose any company that works out how to increase revenue while upsetting and alienating it's users could possibly implement said ideas. But if you're smart enough to do that, I would guess you would also be pretty successful at the traditional approach of increasing revenue by pleasing your users?


I often heard that google's customers are its advertisers, and I want to ask why the users aren't considered end-customers since they and their interaction with google provides the majority of the MONEY that feeds the ecosystem that feeds google?


Google's users are the product, the customers are the advertisers.


So in this scenario, my competitors can spam links for my website and put me into trouble. Google will notice that my website has some spammy links (which my competitor created), and then penalize me for it. Then I have to contact every website that links to me to ask them to remove links to my website... else their website would get penalized too. I get such emails every week.

Rather than these fear mongering tactics, why can't Google just ignore spammy links instead of penalizing the whole websites and all the websites that link to them?


I didn't really like matts answers, some of them are not answers at all.

But! You may have hit the nail on the head. if i had to guess i would say that storing exceptions means that suddenly the algorithm had to check part of the corpus for these exceptions. at the scale at which google operates this could probably lead to rather big size and speed implications.

But maybe i'm just fantasizing.


When monopolies distort markets, the impact of those distortions have outsized impacts on the rest of the ecosystem.

Ask Google engineers how they feel about centralization of power in telecom or finance or politics & they will tell you it is the worst thing ever http://www.youtube.com/watch?v=Ik1AK56FtVc

Yet that same sort of centralization is fine when it is Google.

Those same engineers will admit that any algorithm has some level of false positives & false negatives. They may try to minimize these, but they can't make them zero.

The other thing which gets very little coverage, but is crucially important is that in spite of already having a monopoly market position Google keeps buying search marketshare with: their secret Android partner contracts, Firefox default search placement, iOS default search placement, Flash embedded into Chrome (so security updates happen in the background without sending you to a download site where trash can be bundled in with the update) & Flash security updates that hit all other web browsers coming bundled with Chrome bundleware which sets Chrome as your default browser, etc.

If Google decides they don't like you (for any reason), then you need searchers to use a different search engine in a browser which isn't paid off by Google for default placement. And you need to hope that users are savvy enough to repeatedly say "no thank you" to the automatically bundled Chrome install with their frequent Flash updates.

Google mentions how anything is only a click away or similar, but most people tend to use defaults.

When someone changes those defaults on Chrome users, that's a horrible user experience & the "hijacked" settings must be reset. http://chrome.blogspot.com/2014/01/clean-up-your-hijacked-se... But when those defaults are paid for by Google or changed through Google bundleware it is "a great user experience."

If Google believes their marketshare is a reflection of their superior search offering, they are welcome to stop buying default search placement in other web browsers & stop bundling Chrome installs on Flash security updates. But currently they are spending north of a billion Dollars a year on these activities, which indicates they clearly feel there is significant value to them.


Google's algorithm already analyze if a link is useful or spam. They can simply ignore the links that are spam and count only good links. But, Google decides to punish the websites caught in the fire. If your website has a link Google does not like, Google can penalize your website along with all the websites that link to you or you link to them.


> why can't Google just ignore spammy links instead of penalizing the whole websites and all the websites that link to them

You are suggesting an approach where there are no downsides to link spam. Obviously, players of this game would then ramp up their link spam bots, and run them non-stop. Because it causes no damage to the site they want to rank.

Link spam isn't just a search engine indexing problem, it is also a detriment to the general Web user too. It makes the Web worse, not just search.


That would be a Joe Job, right?


Your comments on this post are a great demonstration of the attitude that is driving me off Google properties, as quickly as I can find viable replacements.


Best of luck staying off of Google properties. It gets more challenging by the day. http://mako.cc/copyrighteous/google-has-most-of-my-email-bec...

Don't like YouTube's copyright claim issues on video game playthroughs? Use Twitch. Oh wait, what's that? Google is trying to buy Twitch. http://kotaku.com/report-youtube-is-buying-twitch-for-1-bill... Oooops.


If manually disavowing a link truly has the same effect as getting it removed by contacting the third-party site's webmaster, then why would people still ask for links to be removed? Is disavowal somehow penalized, i.e. it reduces but does not eliminate the penalty induced if no action had been taken? Or is Google's line that the penalty would be eliminated, but people don't trust that Google's algorithms work the way it says they do, and still ask for the takedown to be safe?


Links which are nofollowed are still seen by users & can still impact that user's experience with the website. Yet those are fine, even if they point at spam, because they don't have a significant impact on Google.

So then what's the difference between disavows and having the links removed? Getting the links removed costs far more time & money. And it gives Google a data mining stream of feedback they can leverage to dish out further penalties.

Google requires some of the links to be removed not because it improves the web, but rather their goals & interests are aligned with punishment. They want to add cost & uncertainty to SEO in order to discourage investment in SEO from entities not formally connected to Google.


> So then what's the difference between disavows and having the links removed? Getting the links removed costs far more time & money. And it gives Google a data mining stream of feedback they can leverage to dish out further penalties.

That's a good thing right? Pushing sources of bad links down through the basement, effectively nuking the source/network from the link graph.

You know Google's ranking system is based on information, and adding more information leads to better results for the Web user.


>That's a good thing right?<

That really depends on your perspective.

If it were impossible for others to throw trashy links at your site in bulk for cheap it could perhaps be a good thing.

If Google didn't change policies overnight & then retroactively penalize you for things which were fine in the past http://searchengineland.com/guest-post-google-penalty-187707 then it could perhaps be a good thing.

Unfortunately, in reality, both of those ifs are hypothetical & untrue. Which means it is absolutely not a good thing.

>You know Google's ranking system is based on information, and adding more information leads to better results for the Web user.<

Quality of information matters as much as the volume of information. Look at Demand Media's current stock price for an example of this.

But in terms of links, you have to think through the impacts here...

Yes if Google freezes some activities they dislike then perhaps that is a net positive for relevancy, however the more they fear-monger about links the less natural linking goes on. Most of the major social sites put nofollow on almost all external links. And with the sea of fear approach to relevancy, in some cases magazines or newspapers will profile a person & not link to the source in spite of the entire article being about that source. I've spent many hours being cited by journalists where in spite of being good enough to be the primary source for their content, there was no link citation.

paraphrasing David Naylor's excellent recent video http://youtu.be/UdflGrCJxYc?t=1m50s "More and more high profile websites are having a no linking policy. Which really is kind of weird isn't it, because if that is a journalistic website and they are writing about your website surely those are the kinds of links Google would want to see. ... It seems kind of weird that the links Google actually wants you to get are the first links to dry up on the web." - David Naylor

The other important factor is the quality of the information created by those who are trying to do disavows and link removals. When people are irrationally driven by fear while in a harmed state, they are NOT acting rationally. Business owners who are selling off assets, firing employees, stalling with creditors, aligning predatory lending to try to keep things afloat, etc. are stressed out & are likely to make many poor decisions in that rushed & panicked state.

Thus some of that more information which is created is junk misinformation.

Consider this email: http://www.seobook.com/images/link-unremoval.png They asked the website to remove the link claiming it was unnatural, and then they emailed again to ask the website to ignore the removal request.

They don't know which links are "unnatural" and so they are using automation to try to sort through it all. Are their automated quick guesses (which are often driven by tools) more useful than all of Google's internal reviews & ratings data which has been built up for over a decade? Color me skeptical.

There's another factor with the disavow data as well. Look at this removal request http://www.seobook.com/images/trampolines.png Our site has about 20,000 unique linking domains referencing it. However over the years we have had well over a million registered user profiles. If only 2% of the registered user profiles were ignorant spammers who spammed our profile pages and then later added our site to a disavow file, we would have more people voting against our site than we have voting for it. And those profile pages were already not indexed & the links were nofollowed anyhow. Those pages are effectively outside the search game, yet those ignorant spammers can still create negative votes against our site based on the fear-mongering.

And as bad as that sounds, there's no reward for actually removing the bad links either. At some SEO conferences numerous SEO experts have gave the advice to "just disavow them anyhow" even if the links are removed. Thus this deluge of email spam offers publishers no value whatsoever, just sunk cost & wasted time - time which could have been spent creating useful information.


If my website is getting spammy links, which I did not create. Additionally, it is out of my control to create / edit / remove those links.

In such scenario, how does Google determines if I created the links? Does Google have precognitive abilities or it can spy on my communication channels (Gmail, Google Voice, Hangouts etc.) to figure out if I am innocent?


That's where the Google disavow links tool comes in: https://www.google.com/webmasters/tools/disavow-links-main?p...

A model is only as good as the information in it. In this case providing details of these spammy links helps you, and every other site targeted through the same sites/networks/fingerprints.

Consider it a reverse no-follow mechanism.


There is no spam, you algorithm is wrong. Wrong because it's implying the link isn't mean to be there.

It appears after years and millions spent it is impossible to get the perfect search algorithm.


"You should never know in advance a link is coming, or where it’s coming from. If you do, that’s the wrong path."

That is an absolutely ridiculous rule of thumb to follow. What exactly is wrong about doing something like:

1) Having a good article on your site about a certain topic 2) Reaching out to other site owners who have similar content and asking them to link to you as an additional resource 3) Getting that link.

Can you honestly, with a straight face, tell me that's BAD!?


Can I, the potential reader, chime in? I hate this tactic because it results in lower quality content getting further up the search rankings. If I'm a gardener, I want the blogs of well-respected gardeners who have real-life community standing and mutual respect to link to each other. What I don't want is two amateur gardeners with crappy writing skills who have made side-deals to link to each other getting on the first page of a Google search.

The situation you mentioned can start out as innocuous but quickly become downright shady. Hmm, you're asking them to link to you, but what do they get in return? Oh, here's a thought, you tell them you'll link to them as well if they link to you, that way you'll both get a ranking bump! Totally fair and win-win for everyone except for the people who are trying to get quality information.


If I'm a gardener, I want the blogs of well-respected gardeners who have real-life community standing and mutual respect to link to each other. What I don't want is two amateur gardeners with crappy writing skills who have made side-deals to link to each other getting on the first page of a Google search.

That might be the ideal result, but it's hard to see how anyone could achieve it with a purely automated system. To make the kinds of distinction you're describing, in general you probably need some form of manual curation by genuine experts with uncontroversial opinions, assuming such people even exist in the field of interest.


As long as we are concerned about the organic structure of markets...

How do you feel about Google sponsoring educational seminars for market regulators where they do not disclose their sponsorship, actively conceal it, and request the entity putting on the sponsorship do the same thing?

http://www.washingtonpost.com/politics/how-google-is-transfo...

It is also worth mentioning that when Google was lobbying against regulation they indeed DID control & manipulate the placement of favorable media masquerading as regular content:

"the staff and professors at GMU’s law center were in regular contact with Google executives, who supplied them with the company’s arguments against antitrust action and helped them get favorable op-ed pieces published, according to the documents obtained by The Post."

---

Another thing with the line of thinking based on pureness...should companies which have repeatedly been caught rigging actual physical markets (e.g. municipal bonds, California energy, LIBOR, interest rate swaps, forex, etc.) in the real world get a penalty by Google for it?

Or is it reasonable that they take their outsized gains from their market rigging behavior & invest some small portion of it into buying out smaller competitors, lobbying & writing regulations to harm smaller competitors, buy some feel good brand ads & do various charity donations to paint a picture of themselves?

---

I also want to quote the following past statement from a Google executive about the AdWords ads Google puts directly in the search results:

"As with all of our policies, we do not verify what these sites actually do, only what they claim to do."

http://www.washingtonpost.com/politics/google-faces-new-pres...

---

and, some icing on the cake, ... ;)

check out the keyword rich deep links to some of Google's affiliate offers on BeatThatQuote.com (for credit cards, car insurance, etc.)

Google bought BTQ & then put a bunch of cross-site keyword rich deep links on it. is that natural? how is that any different than buying links?


dchuk, that's a direct quote from Bing, not Google: http://www.bing.com/blogs/site_blogs/b/webmaster/archive/201... It's in the "Links are all I need" section. Here's the full quote:

"While important as a vote of confidence for the content they point to, there is simply so much link spam these days that it’s tough to know where to turn. Obviously buying links is a dead end, and it doesn’t matter how you split this hair: sharing, encouraging, incentivizing, buying – it’s all the same. You want links to surprise you. You should never know in advance a link is coming, or where it’s coming from. If you do, that’s the wrong path. Links are part of the bigger picture. You want them, but you want them to be natural. If an engine sees you growing tem naturally, you’re rewarded with rankings. If they see you growing them unnaturally, you’re rewarded with penalties."

If you disagree with that, you'd need to take it up with Bing's webmaster outreach team.


Holy moly man, you were the one who provided that link: "It's not just Google that recommends to avoid link spamming. For example, at http://www.seroundtable.com/bing-links-knowledge-18535.html you can read where Bing said "You should never know in advance a link is coming, or where it’s coming from. If you do, that’s the wrong path."

Your statement implies both Google and Bing share the same sentiment.

So now, my question still stands: What about that scenario that I laid out is actually bad/spammy/evil? Why should anyone ever be punished for doing that?


Why shouldn't I ask my customers to provide feedback on Internet? A business must encourage their customers to spread good word by various channels like social media, word of mouth, blogs, links, reviews, invite friends etc.

This practice has existing long before Internet, and internet has made it easier than ever. Google is essentially telling us not to ask our customers to evangelize for us.

Why such hypocrisy? How about the invites system Google uses for many products including Gmail where users are encouraged and incentivized to invite friends.

According to Google's terms if I link to my Dropbox referral link on my blog, Dropbox should be penalized for it because they incentivize users to do that.

The internet should not work according to Google's algorithm and policies, it should be the other way around.


Great point!

Every marketing seminar I have ever been to has emphasised that as much as you can turn your clients into fans, in the end you still need to ask them for a testimonial and that you need to tell them that you love referrals.


In other words, get your links by using a PR firm, not an SEO one,


I'm sorry Matt, but that is a pretty silly thing to quote. After all, what's the point of building a strategic network, or making friends, or any other long, grueling white-hat tactic that takes forever to pull off?

Under that same logic, Google Adwords should never know in advance where their ad clients are coming from, and if you have someone out there trying to on-board big clients (or increase their spend) that's the wrong path.

You say "if they get caught" when sometimes, there is nothing to catch. You're dinging people who have grinded to forge relationships, in the name of trying to keep spammers out of the index.

It makes it too easy for me to hire a spam company to blast an overwhelming amount of crappy links at my competitors who may or may not be well equipped to disavow those links.

Which, if my company has the budget to hire someone to monitor & disavow, and my competition does not, well... somebody's still gaming the system...just not in the same way...


But in this case they didn't spam. The op's site linked naturally to that site, and that site's admin got a message from Google that the op's link is unnatural (when it was natural).

That's what's wrong with this picture. The link was perfectly valid and yet Google recommended that site's admin to disavow the natural link, simply because Google's algorithm isn't good enough to see what's natural and what isn't. So his suggestion is: if it's not good enough, then don't recommend the wrong stuff to people.


higherpurpose, nowhere in the original article did it say that Google said the link was bad. This was a request from a random site (we don't know which one, since the post dropped that detail), and the op can certainly ignore the link removal request.


> Apparently Google convinced them, via their Webmaster Tools portal, that the link looked “unnatural”, and that they should use the Link Disavow Tool to discredit the link.

This is pretty typical of 2014 Google.

To be honest these days I pretty much regard Google as one of the banes of my existence, you guys hold so much power and influence so much of the internet the rest of us have to work around you, I'm currently pulling all our customers off Google Apps for Enterprise for a multitude of reasons (buggy code, broken imap implementations, impossible to get an actual human answer) as well as moving myself away from all Google services (including Android).

I remember when you guys where the good guys.


Exactly. Instead Google should suggest that they investigate the link and appeal the way it was categorized if the search engine was wrong in its classification. Anything less is tampering with the natural way links should work.


Yes, truly Google's rules held no sway over the SEO world until 2014.


That's not what he's saying, and you know it.


It's exactly what he's saying.

People have not only been complaining about this since Penguin came out (around two years now), before that they were complaining about the opposite: that Google's arbitrary rules were creating spammy links all over the internet. It's completely ignoring the past to pretend this is some new thing to fit a preconceived narrative.

Most of the SEO folks that seem to come out of the woodwork for these threads would agree with me that "This is pretty typical of 2014 Google" ignores years of them complaining about the exact same thing, except they're just happy to have more people on their side.


For years we begged Google to simply stop counting links they didn't like. These included generic directory link, crappy blog links, article farm links and on and on and on. They claimed to be able to tell what was good and what was bad yet the search results rarely showed that in certain niches.

Then we stared seeing the first indications that real "negative" SEO was possible in certain circumstances and we complained even further. Stop counting links you don't like. Now we are at a point where we see old websites being told to remove or disavow links that may be as many as 10 years old.

Can Matt or Google really stand there and say that it took them 10 YEARS to figure out that not only was that a link that they didn't like, but it was so evil that now it needed to be removed from the internet? The solution has been, and always will be, stop letting things you don't like count as a plus in your algorithm. When in doubt, don't count it.

Instead, we have Google adding thousands of webmasters and site owners as a living part of the algorithm and getting them to remove all those links, plus many they can't see or understand the value of, by the use of fear and scare tactics. Remove these links are we will ruin your website rankings. It doesn't matter if you created or asked for them or not, remove them.


> higherpurpose, nowhere in the original article did it say that Google said the link was bad

Yes, it most certainly did!

From the article:

> Apparently Google convinced them, via their [read: Google's] Webmaster Tools portal, that the link looked “unnatural”, and that they should use the [Google] Link Disavow Tool to discredit the link.

(emphasis added)


You emphasized Google, but you should have emphasized apparently, because nowhere in the actual email is the OP's conspiracy theory about Google mentioned (or even the word Google!). It's quite likely they noticed a penalty and are just going through all their back links to find shady content.


Sure, this is all hearsay and second- and third-hand information, and the article is really sparse on details.

> It's quite likely they noticed a penalty...

Yes, and where would they have noticed that penalty?


> Yes, and where would they have noticed that penalty?

In a drop in incoming traffic, so with whatever they're using for analytics.

In all honesty it's probably just a new SEO consultant trying to clean up past sins and move on with whatever the latest grey hat strategy is.

IMO it's a gross overreaction to a form letter that a high percentage of webmasters have received.


Way to avoid the point. And to be honest, the whole oblivious act is getting ridiculous at this point. Don't post if you are going to play dumb. In case you're really that dense, a webmaster got a message FROM GOOGLE, that a perfectly legit link was spam. The provided email is their response to that.

The whole point is, Google spreads FUD to paper over its inadequate algorithm. People believe and look up to you as a source of truth. However, you take that trust, and you ask people to do your dirty work. That means more work for them, and not every small business has all sorts extra time money to spend on stuff like that. This is not to mention that there are probably a bunch of other more important things they could be doing. (and I know you know this, your friendly face is the head of FUD over at Google)

With all of your power and authority, you have a responsibility be more honest and straightforward. Not everyone knows better, and when people end up listening to your Google/self serving advice, they are ruining the open and free nature of the internet.

This wouldn't make me so salty, but the complete BS and favoritism exhibited by you and Google is obnoxious. So, sometimes, it might be better if you just shut up.

sorry if there are any typos etc, my (virtual) keyboard is spazzing out.


They're not talking to you about link spamming.


agreed, matt is talking in generalities at this point


Matt is not reading what the discussion is about at this point.


More specifically, I believe people feel they aren't being heard at this point. Rationalizations are the tell.


It's both Google trying to force web sites to help their algorithm and spammy sites trying to exploit the current algorithm that are causing these annoyances.

I suspect neither would be a major issue if Google modernized their algorithm to take into account (mostly/only) which links people actually clicked on rather than what is linked from where. They'd only have to be able to distinguish people from click-bots, which might be easier to do with some confidence than recognizing the intent behind a particular link (spam/SEO or real information for visitors) - I hope they are already able to do this, for the sake of AdWords customers ...


> The model (google's algo) should fit reality. Rather than forcing reality to fit the model.

Google's algorithms should reward behaviour that's best for Web users, and penalise behaviour that is worse for Web users.

That way, for people who want to game search engine rankings, they are pushed more towards techniques that benefit web users in the course of trying to benefit themselves.

The problem with models is that they are not accurate simulations of the real world. They have flaws and simplifications, and those give rise to people taking advantages of those flaws. All models and representations have flaws.


The Internet would naturally be a tragedy of the commons - and entropy fighting is something Google seems to be doing well.


> The model (google's algo) should fit reality. Rather than forcing reality to fit the model.

It is impossible for Google to exist at its current scale and not influence the Internet.

The tiger is born out of adaptation to the jungle, but in time, the jungle too will adapt to the tiger.


"If you read the original post closely, it's clear that this is a site asking for a link to be removed--the quoted email isn't from Google."

Matt, this is beyond disingenuous. If Bing or Yahoo responded in the same way, I doubt we would hear a ringing endorsement from Google on the subject.

You've indicated in other comments that basically your algorithm is everyone else's problem to deal with. You can use whatever words you like, but the actions are what matter.

At this point, it sounds more as if Google is concerned with "being right" more so than "doing right".


Now now, I think you're the one being disingenuous. Google's algorithm is not "everyone elses problem". If someone doesn't care about what appears on search result pages, they can just ignore everything search engines say: all their guidelines, all their advice, just chuck it out the window. Your website will still work fine.

This blog does not indicate some fundamental breaking of the web. It indicates a guy got a random email from a random website, that he is free to ignore if he wants (and in fact should ignore, because the webmasters in question don't even need to contact him, they can use the disavow tool).

No work to do? No problem!


> It indicates a guy got a random email from a random website, that he is free to ignore if he wants

It's reasonably clear to me this email was anything but random. Take Google out of the picture and this communication never happens. Both the sender and receiver are taking actions with consideration of one single point -- how does this affect my Google search ranking?

If Google weren't the literal default entryway into the Internet, your argument would be valid. There is precedence in this scenario as well. Back in the 90s, many people said the same thing regarding Microsoft and attaining placement on the Windows desktop for new shipping PCs. This was a key element of the DOJ trial.

The argument that a powerful intermediary has no effect on the conduct of others is simply choosing to ignore certain facts.


Your thesis would be true if Google didn't have monopoly marketshare & continue buying default (search / browser) placements in other software.


> If you read the original post closely, it's clear that this is a site asking for a link to be removed--the quoted email isn't from Google.

Did you read the article? The reason that site is asking for the links to be taken down is that they believe (apparently based on feedback from the Webmaster Tools) that it will please the Google Gods.

Spammy links and irrational fear of spammy links are both unnatural linking behaviors that result when sites are more concerned with pleasing search engine algorithms than with pleasing human users. That's what the article is talking about.


"Spammy links ... are both unnatural linking behaviors that result when sites are more concerned with pleasing search engine algorithms"

ForHackernews, we say in our quality guidelines to "Make pages primarily for users, not for search engines." If the SEO whose email was quoted in the article had followed that advice, they wouldn't be in the position of trying to clean up spammy links after getting caught spamming.


>ForHackernews, we say in our quality guidelines to "Make pages primarily for users, not for search engines."

The problem with that is that it's not true (in my admittedly very limited experience). I used to believe that until I created a small website and found it wasn't getting listed in Google and I had to try and guess why and carefully read the rules.

Making pages primarily for users means you unthinkingly do things like having duplicate content on multiple pages/subdomains/sites or linking to a site that reviewed your product after you gave them a free sample (a paid link).

I'm sure to any even vaguely experienced webdev they know not to do these kinds of things (use canonical links, nofollow and all that jazz). Somebody just throwing a few pages together who has no experience in this stuff easily falls foul of the rules. The rules may seem obvious to people who understand how search engines work. If you have never heard of pagerank and don't have an understanding of how search engines work and how people try to game and spam search engines, the rules are not obvious at all and certainly the rules are not the same as the natural behaviour of somebody creating a legitimate site with the users in mind.


How is duplicate content made "for users"? Why would I want to find exactly the same prose in multiple places?


* Paginated content with different filters or sorting? Almost the same content on multiple pages

* Product with a category name in url, or without, or under multiple categories

Both are very useful for humans, but have to be massaged with canonical urls to please Google machines


I'm reasonably sure Google is familiar with pagination and filtering, seeing as that's pretty much the entire thing their own website does.

I don't see how the same product under a variety of URLs, one for each category it's in, is at all useful to me as a human. It actually rather annoys me when the same page has a whole bunch of URLs: I can't tell which is the "real" one, my address bar frecency gets messed up, visited links don't work correctly, etc.



Maybe you run a store where you sell Widget and Deluxe Widget. The only difference is Deluxe Widget has a higher max RPM, but the operating instructions for Widget and Deluxe Widget are identical.

As a consumer, if I came to the website looking for instructions on how to use my [Deluxe] Widget, I would reasonably expect to find the same operating guide on both product pages.


Maybe you should collapse these admittedly identical items into variations of the same product, which will reduce the useless duplicated clutter I'd experience when searching your catalogue.


So are you saying that "clean up" and suspicious link notices only happen when a site hits a threshold. A state machine of sorts?

Frankly, I think "spammy link penalties" is an approach that cannot work long term, any negative weight put on links is too easy to game by competitors.


Matt, what's to stop people creating a website (or multiple websites) filled with spammy links pointing to their competitors websites, thereby reducing the weighting of a competitor in Google's pagerank algorithm. Is there protection against that sort of activity?


I think this is pretty common, if you want a website to rank. Then rank it as best as you can then smash the other websites with shitty, unnatural links. Pretty good idea actually but if it continues then it creates a really bad environment for search engines.


Not just a good idea this is actually done already by the blackhat guys, they also use the same technique to extort you in the classic "nice website rankings you have here, be a shame if anything happened to them" manner.


The disavowal tool really isn't that hard to use. Yes, it is more work than no work, but if you depend on a search engine for traffic, you're going to have to do some work to optimize for it (like, sign up for webmaster tools).


There is no protection against this from a publisher perspective.

The vast majority of link removal requests I've received do not come from an email address that matches the destination of the link for which the removal is requested.


they just re-hire the same SEO guy who bought all of the spam links in the first place. :)


Not the case. I do speak to a lot of the smaller merchants who email as I feel for them.

They still don't really understand the web, and they hired someone who appeared professional to them (and probably cold called them). Only after a while do they realise they're paying a lot of money for something unquantifiable and with no guarantee of hitting the front-page of Google, and so they end that relationship (in fear, as they're told that whatever placement they now have may be in jeopardy).

So starts the merry-go-round, where the smaller merchants constantly jump from one bad SEO company to another. Each one creating mess, whilst claiming they'll clean up the mess before.

I do feel for the smaller merchants. Their business is being eaten by the internet, and they're missing out and have ambition but are being fleeced by sharks every step of the way.


Yes, we work to avoid this sort of attack when designing our ranking algorithms.


Can you give any specifics? Unless I'm too stupid to figure it out, mathematically, the only possible way to do that is simply ignoring (potentially) spammy links, instead of considering them negatively. I mean, SEO linkfarm --> publisher site looks the same as Competitor's linkfarm --> publisher site.


For most of your comments are getting greyed out, I might think that there is some other people that'd back my words.

You're Google employee, right? You lot have gone quite cheeky nowadays. You're exploiting the monopoly you hold in the search market, to make people adhere to your standards, instead of adhering the standards the internet community naturally generates. I'd suggest Google engineers to go develop better algorithms instead of telling the internet off.


"If you’ve done as much work as you can to remove spammy or low-quality links from the web, and are unable to make further progress on getting the links taken down, you can disavow the remaining links. In other words, you can ask Google not to take certain links into account when assessing your site."

That's taken straight from Google's webmaster support forums... https://support.google.com/webmasters/answer/2648487?hl=en

Even in situation number #1, Google clearly encourages webmasters to "do as much work as you can to remove spammy and low quality links..."

I happen to disagree with the article, but it's kinds disingenuous to claim that Google had nothing to do with that link removal request.


The original article dropped all the specific details like the link and the company that requested it. I'll certainly dig into this specific situation more soon, but I didn't want to pull engineers to look into this specific situation over the weekend.


Matt - I would be happy to share any details about this specific situation. I removed the links from my post because I find it poor taste to reprint details from personal emails for public consumption. I'm on Twitter @jeremypalmer.


I don't really see any reason to know the specifics or to investigate further. The authors point was simply that lots of organic links are requested to be removed to play it safe.

I don't think Google needs to spend resources to turn around and tell this guy, "got'ya"...

Personally, I don't see the harm in having organic links being asked to be removed. I would never remove a link that sends referral traffic, and I see no reason to keep a link that doesn't send me visitors. I have no problem having someone no follow a link that sends me traffic, because it will send me the same traffic...

so I think the issue here is less about actually breaking the internet and more about how it makes people feel when they thought they were helping you by linking, and then they get accused of destroying your rankings by asking for the link to be removed....


Situation #1 happens fairly regularly - do you have/can you do a video that can be sent to the company asking for links to be removed which clearly explains what is happening? We have had clients such as a small bathroom fitting company where the company that made the showers that they fit demanded that our client remove all links to them. There was a degree of bad feeling, as our client fit a lot of showers from that company - the links were because they genuinely were endorsing them.

A video from Google would be useful to help explain what is happening here for all parties involved.


We have made a video for sites that received a notice for unnatural links: http://www.youtube.com/watch?v=6UFHCw4Q_FQ

I thought I'd made at least one video for sites that have been getting link removal requests. I'll check, and if we haven't made one then I'd agree that would be useful. Thanks for the suggestion.


I guess that's kind of beneficial for Google though - now if people want to find out more about the showers your client fits, they have to Google for them, and Google can sell lots of profitable advertising space on those searches to competing bathroom fitting companies. I wonder how often it's in Google's financial interest to break hyperlinking on the web?


I have been complaining of this problem for a few years now and it's getting worse. I now receive several new "link removal" requests per day (I receive many more repeats of old removal requests).

Yes, Google isn't sending these so you could argue that Google are not directly to blame.

But, the merchants in question who are sending them are doing so because Google has made them believe they are (or will be) subject to some penalty if these links are not cleaned up.

I run forums and all of our links are natural links. For the first year I would look at every link that was asking to be removed and found that, without a single exception, every link was created by a trusted person in the community helping to answer a question by someone else. Every link I checked was natural and high quality, none... literally none... were suspicious or spam.

It may be that these merchants did involve SEO crews who did create a lot of spam links, but those are not on our site.

What's more worrying is that merchants feel so overwhelmed (numerically there are a lot of links to deal with) and in such a state of panic about this (they fear a bad placement could have significant effect on their business) that now they're hiring people to clean up SEO mess, and those people create 4 problems:

1) They have no clue which links were the result of SEO, so they just request every link they can to be removed.

2) They send these removal requests from emails like someone@web-marketing-group.org.uk when the links point at www.watches2u.com so it is impossible to verify as a publisher that the person requesting link removal represents the merchant to be de-linked (and #1 means that they don't enter into dialog as they can't, they're sending tens of thousands of requests).

3) The removal notices seldom have enough info to be acted on. One forum I run has several million posts, and the removal request may simply be "You've linked to blah.com, please remove it". OK, more verbose and almost identical to the OP's example, but not a useful pointer to the page or comment that contained the link.

4) If, as a publisher, one fails to remove a link, then their automated removal request scripts will now spam the publisher routinely with the same request.

If you can finally get a conversation that says "Do you represent the merchant, have you got a full URL to where the link exists on our site, and are you sure this isn't a natural and good link?", then every merchant has so far agreed that the link should stay.

But... for all of that, the effect is very real for the publisher: We get spammed evermore, from unaccountable sources that we don't know has a link to the merchant, asking for links to be removed that are actually very good quality and with good context and content.

It's a mess. A mess for which the merchants may be to blame (for originally doing SEO, and now for handling removals so badly), but this mess is definitely due to Google creating the situation.

My policy now: I put link removal requests in my spam folder.

That is how I regard it, this is just a new type of spam that Google has created. We never engaged in SEO or supported anyone doing so, we took an aggressive stance against spam, but as we have content that links elsewhere we are being spammed constantly with these requests and we carry the burden of the problem.

PS: The example cited above was just the top one in my spam folder, the link can be found on this post in the form of an img tag: http://www.lfgss.com/thread17604-52.html#post2119173 , it's the photo of a watch posted on a horology thread within a cycling forum by a long-standing member of the community with high karma. The request for link removal did come from an @web-marketing-group.org.uk address. The request will be ignored. We don't do "nofollow", but it's irrelevant as that isn't valid on img tags.


It's not a huge surprise that some sites which went way too far spamming for links will sometimes go overboard when it's necessary to clean the spammy links up.

The main thing I'd recommend for a site owner who gets a fairly large number of link removal requests is to ask "Do these requests indicate a larger issue with my site?" For example, if you run a forum and it's trivially easy for blackhat SEOs to register for your forum and drop a link on the user profile page, then that's a loophole that you probably want to close.

But if the links actually look organic to you or you're confident that your site is high-quality or doesn't have those sorts of loopholes, you can safely ignore these requests unless you're feeling helpful.


> For example, if you run a forum and it's trivially easy for blackhat SEOs to register for your forum and drop a link on the user profile page, then that's a loophole that you probably want to close.

It isn't, and never has been.

We operate a nursery period ( https://www.lfgss.com/thread9350.html ) in which the actions of new members are both severely restricted and observed by existing members, and all initial content must be 100% public and observable by members (no private messages, no user profile customisations).

We very aggressively remove content and ban anyone who creates any content that appears spammy and continue to encourage reports of spam even long after they have been registered.

We report those individuals to http://www.stopforumspam.com/ at the same time preventing signups from anyone that SFS reports as coming from an IP recently used to spam.

Comments are run through the Akismet API and moderated if they are flagged as spam.

When a spammer is caught (a few times a year someone does get past all of this - for a very brief moment), then every piece of content of theirs, and any other content from nearby IPs is reviewed and usually removed.

We made profile pages non-public, we even removed the customised profile fields from display (though we do leave them in the form as it helps us spot any scripts that are trying to spam us and we autoban them).

To put it mildly, we are aggressive.

We've made a really poor experience for new members of our community to help ensure the quality of the content on the site and that we're free from spam. We felt that was worth it from day one, we still feel that it's worth it.

Which is why it's hard to feel good about Google when our reward for fighting spam on our corner of the internet is to be spammed by merchants whipped into a state of panic by Google.


buro9, this is a great example of a well-maintained forum, and as such I agree that you're doing the right thing by ignoring link removal requests.

Unfortunately, if you look at wider area of forums as a whole, a very large fraction of them are unmaintained, unpatched, or otherwise not kept safe from drive-by spamming. There are entire forum spamming software packages that will register spam users for dozens of different forums in order to drop spammy links. You can even lease the spamming software by the month. :(


Also... what happens when, as a consequence of us ignoring requests, Google decide that a site with a high number of disavowed links is a spammy site?

I figure already that you'll eventually consider that a valid signal, but I'm sure you could already check my site and that there must already be some disavowed links, yet you personally admit this is a well-maintained forum.

There is a fear that we are being caught in the crossfire, and it's hard not to believe that quite a few of us are going to get hit by that crossfire.


Thanks, and I agree. But... it doesn't resolve the problem that publishers who always fought spam, who stayed as far from the SEO battles as possible, and who have largely spam-free sites, are now subject to a constant and increasing flow of link removal requests as a consequence of Google's pushback against the merchants who did use the more spammy SEO tactics.


except when he is disavowed as all the removal emails say they will do.

I've added rel="nofollow" as good-enough when trying to help out people throwing too-far of a net.


And there it is again, the point that the author of the article is making . .

"Do these requests indicate a larger issue with my site?"

When the truth of the matter is that it's Google's issue. Why does the burden of the cost have to be on the publisher?

This is the 3rd time someone is pointing this out, and it pretty much showcases "Google"'s attitude on this.

At this point the cost of becoming a good source of information has increased greatly to the point that very few publishers with resources can actively manage all the rules and "guidelines" Google decides to throw at the content creators that provide the content that makes the search engine valuable.

I see this everyday in our business as smaller but richer sources of content just disappear on many topics. Blogs, forums and independent enthusiast content sources.

Google has created a one sided ecosystem. That is consolidating traffic to so called "good" sites.

As a long time publisher for a large media company I think it's time to take the internet back from Google. Who care's what Google thinks and does and start building for a better internet.


I've read the thread up to here, and I do think Google is to blame for some heavy-handedness and some rather tone-deaf spokespeople (not that Matt_Cutts ever identified himself as speaking for Google, though he acts like it). But I think the bigger point that everybody is missing is that the Internet has become awash in all sorts of information, "good" and "bad," well-curated and spammy, human- and machine-generated, white-hat and black-hat. Just because YOU post information, and post a blog article about how it is good information, doesn't make it so or not so to an objective observer.

Google is what it is: trying to sift through all that information algorithmically and using some forms of AI so it can extract what humans think is good information. I agree that a good forum that has good policies is getting hurt by some of google's actions, but it's the larger system, not Google. To make an analogy: almost everybody drives and everybody uses roads, but nobody should say: I'm a good driver or I don't drive, I've never been in an accident, therefore I cannot be in an accident. With all the information and intents behind such information on the Internet, and with google as the self-appointed traffic laws, there is still bound to be some collateral damage, sometimes because of how Google tries to deal with the dreck.

As an aside, one can view Google as a just one way of monetizing information on the Internet, competing and co-existing with many other ways. You can also view it as a giant AI algorithm that gets rewarded (advertising revenue) when it finds the good information that humans are looking for and defends itself against others who have other interests.


What are people "pointing out" exactly? This is a ridiculous argument.

If you don't care about what Google's view of the internet looks like, then ignore them, and ignore any emails from people asking you to change things on your site for them. Done. Simple.

This issue crops up because people usually do care what Google's view of the internet is and would like to help make it better. Marking links as nofollow and so on is entirely optional: only people who want useful search engines do it, and others don't care.


Is HN an example of a site that makes it trivially easy to sign up and drop a link on the user profile page?


Yes, we get tons of spams on our blogs and forums.

Thanks to Google for creating these parasites. The link-spam problem has been caused by Google in the first place.

I don't remember getting such spam in a pre-Google Internet world.

Small website owners like us may not have the technology and human resources to combat such spam. Therefore, we are caught up in the fire and get penalized by Google.


" "Do these requests indicate a larger issue with my site?" For example, if you run a forum and it's trivially easy for blackhat SEOs to register for your forum and drop a link on the user profile page, then that's a loophole that you probably want to close."

Wouldn't that be like every vBulletin site?


Are you not able to simply ignore bad links because when you did that years ago, people would just pump and dump thousands of links to see what stuck?


I think that a lot of people are missing the point that sites that are being penalized or demoted in the Google search results because of unnatural linking received that penalty because they were cheating.

When webmasters are writing to ask for a link to be removed it's because they believe that that link could possibly be one that was made with the intention of increasing their PageRank and manipulating Google. BUT...sometimes it's hard to tell whether a link was naturally placed or whether it was one that a former SEO made on their behalf. As such, it's certainly possible that an email requesting link removal could be sent out in error. If that's the case, then ignoring the request is fine...or even better, writing back to inform them that the link was not purchased, solicited or self made would be really helpful.


Matt, I am the webmaster of http://TeenageMutantNinjaTurtles.com -- before I took the site over, the forum was hacked (~2010) and we were hit with a ton of bad links (posted on our forum and lots of underage adult websites linking to us).

I've been working the past two years, from contacting all the webmasters via WhoIs information, creating unique content on a weekly basis, having a good social media following, and still for the life of us, we cannot get ranked well organically on Google.

Could you personally look into this or give me any advice on resolving this issue because I feel like I've done everything possible and we put a lot of work into the website over the years. I would really appreciate it.


I have a particularly odd case where neither situation 1 nor situation 2 seemed to happen. Webmaster tools doesn't indicate that there is anything amiss and the pagerank >= 7. We do almost no promotion of our site and have never done link sharing of any sort.

How am I as an outsider of Google able to discern any information about what's going on in this situation? I assume this isn't a unique case but there is literally 0 information. I'd like to clear this up - not just for me but for the community. I know this may be the wrong place to ask (I'm not sure where the "right" place would be), but would you be willing to contact me? caudill m at gmail


> For example, we might say "we believe site-a.com has a problem with spam or inorganic links. An example link is site-b.com/spammy-link.html."

If you are able to identify problem links, you can just ignore them. Or am I missing something?


IME, this sort of letter is pretty much always a spammer trying to get their past comment spam removed.

You're penalising spammers, and it's being effective. GOOD.


Sure you did Matt, we're also seeing how the only good link as far as Google is concerned is an Adwords one. Can you stop insulting our intelligence, we see how every quarter your Adwords is increasing by double digits as web publishers are going broke.

All Google roads and products lead to ad clicks and Adwords. Enjoy it while it lasts.


Hey Matt_Cutts, sorry that this is a little off-topic, I don't really know what would be the correct medium for this... so here I go: could you please kill mypdfmanuals.com from Google hits? I was searching for some instruction manuals (my exactly query was: <pax wardrobe manual filetype:pdf>), and again and again the top hits were unhelpful mypdfmanuals.com links, which don't even have manuals, they just tell you to go a site where you can get the "full version" of instruction manuals that are otherwise publicly available for free.


This doesn't help solve the problem itself (and is an annoying extra step), but as a user you can append '-site:annoyingsite.com' to your query to get rid of results from the offender.


If you use Chrome try the Personal Blocklist extension: https://chrome.google.com/webstore/detail/personal-blocklist...


Google web search has a feedback page.

As you can imagine the process is opaque. I have reported some sub-optimal websites and seen a difference a few weeks later. I have no idea if my comment was read by a human let alone whether any action was taken as a result.


Jeremy Palmer is a SEO marketer, who also appears to be doing some scammy affiliate thing. Google considers his links low quality because they are low quality.


> Google considers his links low quality because they are.

So what? Why should I have to care that Jeremy Palmer (or any other person) is linking to my site?

The point of the article is that webmasters are going around trying to get links to their sites removed because it might get them penalised by Google. That is not natural or desirable behaviour. Site owners shouldn't have to care about that. Many have resorted to adding 'nofollow' to all of their own outbound links for fear of being punished by Google for linking to 'spammy' sites and appearing to be part of a paid/circular links scheme.

It all sounds very broken to me.

Note: I have no interest in SEO whatsoever, and I only have a single low traffic web site.


If you've got bad links, just disavow them. Wikipedia has to nofollow links because anybody in the world can add a spammy link to an article. NYT nofollows links because they're old media, and don't understand the web.


People use nofollow because they believe (rightly or wrongly, none of us know for sure since it's a secret) that if you link to a site that later turns out to be a link farm you can be punished for it because it appears to Google that you are part of the paid links scheme. 'Turns out' meaning either it was a link farm all along and you didn't notice, or didn't think it mattered, or because the site got bought/hacked and now is just a link farm. (I'm using 'link farm' in the broadest possible sense here since I'm not familiar with all the various ways a site can be deemed to be engaging in paid links and I believe there are quite a few and it's intentionally vague).

Even if Google isn't penalising you for that today, they may well tomorrow. So using nofollow seems like quite a rational thing to do for any individual site as there isn't any downside as far as I know, only potential benefits.

Additionally, manually disavowing links hardly seems like a scalable solution. As somebody with a small website that I spend very little time maintaining, I don't want to have to regularly be checking some Google console for 'bad links'that I have to manually disavow to prevent getting delisted.


> Even if Google isn't penalising you for that today, they may well tomorrow. So using nofollow seems like quite a rational thing to do for any individual site as there isn't any downside as far as I know, only potential benefits.

Breaking internet connectivity is a downside for society as a whole.


> The point of the article is that webmasters are going around trying to get links to their sites removed because it might get them penalised by Google.

That's because previously webmasters we going around trying to get links to their sites added because it might get them rewarded by Google.

Google is trying to prevent the aberrant behaviour that the success of Pagerank caused in the structure of the Web, unfortunately there can be some collateral damage.


Yep.

http://www.newspapergrl.com/affiliate-marketing-training-wit...

It's hard to cry when justice is served.


Justice? The people who were afraid of Google's reaction where the ones he linked to, not Jeremy Palmer himself. Even if he is a spammer, how is this justice?


Good point. Maybe Jeremy can get a new gig doing reverse SEO. On peoples competitors. Apparently he already has the cred.


It is really sad that instead of discussing the merit of his argument, we are debating the quality of OP's site.

It is ridiculous that I have to police my site for low quality inbound links or risk getting banned (even more so for mom and pop webmasters who are more susceptible to link penalties). Lastly, if you are penalized, try recovering unless you are big brand like Overstock or JCPenney with millions per year in Adwords spend.


> It is really sad that instead of discussing the merit of his argument, we are debating the quality of OP's site.

No it's not, because his argument is making the assumption that his content is high-quality.


He might have low quality links, but his points are still valid. it's not our perogative to link the way Google us to. We should link however we want, and it is up to Google to figure which is link is good, and which are bad.


> it's not our perogative to link the way Google us to.

Who said it was? Google didn't ask him to do anything. A random person on the internet did. I frequently get emails from random people on the internet who ask all sorts of insanely idiotic things that have little basis in reality.


Did you even read the article?

> Apparently Google convinced them, via their Webmaster Tools portal, that the link looked “unnatural”, and that they should use the Link Disavow Tool to discredit the link.


Yes and that's exactly my point. The email doesn't mention anything about Google, it just says they are cleaning up their links and if it's not a big deal please delete the links on your site. This isn't uncommon, just like it's not uncommon to get emails soliciting links.

The OP has come up with a theory about the backstory and then used that theory to blame Google for ruining the internet (ignoring the fact that the entire "credit" he wants is Google's invention in the first place).


Unless you have a better theory, I would call it an educated guess. "They're just being random" is not a better theory, IMHO.


The most likely situation is that the company who sent the letter hired a shady SEO. That SEO did spammy things that got them penalized. They brought in a new SEO to clean up the mess, and that SEO is trying to undo all the damage the previous one caused. They are trying to remove every link they can find since they didn't do the spamming in the first place and don't know which are causing the problem.


That IS a better theory that "I frequently get emails from random people on the internet who ask all sorts of insanely idiotic things", indeed.


Google doesn't tell you exactly what links to remove when you have an unnatural links penalty. They'll often give you 2-3 examples but then it's up to you to determine which links were made with the intentions of manipulating Google and then asking for removal.


The letter from the company asking to remove the links reveal they used a shady SEO company to generate artificial inbound links. Not a surprise that Google got some false positives -- particularly coming from a SEO blog.

Not really "breaking the internet".


It pretty much is if organic hyperlinks become feared entities.

No hyperlinks = no internet.


Really? It often seems like the original PageRank algorithm was using links as a measure of content quality and relevance, and now everyone claims they are. But thanks to SEO, we saw that was false (homework question: would it still be true in an Internet where all actors are committed to creating "good" content?). I think links are independent of content quality (or rather search result ranking, because quality is subjective), and we need better AI based solely on page content.


Consider the argument the author is making, rather than addressing his person. The argument is:

The tail named Google is wagging the dog named Internet by sending thousands of web masters on a wild goose chase of policing sites outside their direct control, essentially the entire internet. It's a giant waste of human capital, and it only exists due to Google's dominant market position and their willingness to solve problems of their search algorithm at the expense of the third parties.


This alone doesn't mean there is no problem here. A spammer has power, or at least perception thereof, to lower legit site's ranking? Not nice.


I've known Jeremy for years and he's one of the few I consider quite white-hat from my dealings with him.

Please do research before you judge :)


My solution has been to recommend friend and family to switch to http://DuckDuckGo.com (or just http://ddg.gg)

This may seem to be an impossible task, but in days not too long ago people switched search providers often. I went from the curated links of Yahoo, to Lycos, to AltaVista, to Webcrawler, to Google, and now DDG.

Google got a lot of mileage out of my in my mind with their motto "Don't be Evil", because I had some trust for them. I don't trust them anymore, and I regularly explain to others why they should no longer trust them either.


My understanding (and I'm a DDG user) is that DDG largely utilizes other services, and to a large extent Bing, for its search results.

But their own moral authority and index reach are limited.

That said, I still encourage using DDG on both privacy and increasing search engine competition grounds.


https://startpage.com and https://ixquick.com are the alternatives I use.

The former only proxies queries from Google, whereas the latter works on multiple engines.


Using a proxy for Google does absolutely nothing to avoid their judgements of websites. Startpage will not provide you with URLs that Google has banished. As an example compare the following results for "Game of Thrones S04E06" on each site:

https://encrypted.google.com/search?hl=en&q=game%20of%20thro... https://startpage.com/do/search?q=game+of+thrones+s04e06 https://duckduckgo.com/?q=game+of+thrones+s04e06

The first two provide mostly spammy results or subtitles. The DDG results are mostly real sites with torrent or streaming copies of the show available. Which search engine do you think is more useful?


what is the love this crowd has with duckduck?! it is nothing more than bing with EXTRA ads!

also, it display a retarded "this page uses javascript. click here for the content" or something, every time i use it from my HTPC with firefox+noscript.


https://duckduckgo.com/html/ should solve your htpc's problem.


It's about not being tracked while you're searching.


Duckduckgo crawls a fraction of the web that Google does. Would you like a Tomato or a Cherry Tomato?


And Google crawls a fraction of a fraction of the entire internet, last count around 0.04% http://webapps.stackexchange.com/questions/11740/how-much-of...

All I care about are the quality of the results, and for that DDG does a pretty good job.


Irrelevant. Google has an index reds of times bigger than DuckDuckGo. Therefore it's results can be much more accurate. You can't debate that DDG is an inferior engine, it's fact.


DDG has horrible results because it barely scratches the surface of Google's magnitude of results. Sorry Chuck, it's an inferior product


From my understanding Wikipedia adds nofollow to stop incentivising people from constantly trying to sneak links into Wikipedia to boost SEO.

Any ranking system that gave any value to links would suffer this problem.

nofollow is a necessary evil if you allow untrusted sources to publish content on your site that contains links. It's a way of saying "I don't vouch for these links in the same way I vouch for other links on my site".

Sounds pretty reasonable to me.


The issue here is that a link is not supposed to say "I vouch for this site".

A link is supposed to be a hyperlink.

Google creates a false incentive to link/not link, and then complains when people link in a way that they don't approve of. This is a problem of Google's making.


That's correct, I came here to make that comment, whereas he's implying that Wikipedia did it to preserve their own ranking, which is not true.


is google breaking the internet? naah (well at least not with google search - G+, google local and the new horrible google maps is a complete other story)

are webmasters breaking the net because they follow SEO worst practices? definitely.

i wrote this article for techcrunch in 2010 http://techcrunch.com/2010/07/07/startups-linking-to-your-co... the tl;dr: pagerank is thoughtcancer, if you start thinking of links as some flow of mystical pagerankjuice you will make bad decisions, decissions that will hurt your business, your users and in the end the internet.

the stupid "remove link emails" are just the newest iteration of this thoughtcancer.

my recommendation stays the same: link to whatever you like and link to whatever your users like, want or need, oh, and also link to your competition. but for gods, your sanities and the internets sake: don't do it for any kind of page/trust/magic-rank or any kind of link/penguin/panda-juices....

my name is franz enzenhofer / i'm the most successful SEO in europe / i do not care about links


> the tl;dr: pagerank is thoughtcancer, if you start thinking of links as some flow of mystical pagerankjuice you will make bad decisions, decissions that will hurt your business, your users and in the end the internet.

Golden. It's not Google who's breaking the Internet; it's the people who try to game the search engines.


"As a publisher I refuse to nofollow any links, outside of banners and advertisements. I compel you to do the same."

I believe you mean "implore."


Be warned, if you refuse to comply, he might link to you without "nofollow" as punishment!


Maybe he's just really powerful?


Fixed.


It appears to me that Google has a long term ambition to become the sole means of navigating the Internet. Inter-site links are fundamental to topic-related navigation, as the Internet was originally intended. It appears that Google is trying to depreciate inter-site linking in general, and skilled PR misdirection on the topic by a Google employee doesn't reassure.

One has to wonder to what extent the entire phenomenon of link spamming sites might be a Hegelian Dialectic tactic by Google - ie a manufactured problem, intended to prompt a 'solution' that is in fact more beneficial to Google's unstated intent. Are we really to believe that there's no algorithmic solution to mitigating the link-spamming nuisance?

Another dimension of this same ambition, is the movement to obfuscate URLs in browser address bars. Traditionally, Web users could copy, save and manipulate links directly as another means of navigating the Internet and maintaining their own records of Web places. As browsers progress further in the direction of eliminating direct user visibility of true URLs, this navigational method becomes less available. The Google-promoted alternative? Just search via Google!

As for why Google would want to do this, the obvious answer would be the usual 'money and power'. If Google succeeds in virtually eliminating all navigational alternatives to Google-searching, they then own the Net. For instance, if they wanted to make any given web site disappear, they could simply de-index it. That's a politically very dangerous power. Even if Google has no political agenda now (a debatable point), given such power they'd be guaranteed to become political. Power corrupts, etc.

We've been through this before, with a gold ring and a volcano. It's generally a bad idea to create 'one thing to rule them all', and Google is no exception.


> Site owners and publishers are now afraid to link to each other because they don’t know how Google might respond to that link. For example, Wikipedia and the New York Times have added the “nofollow" attribute to most of the links outside of their editorial control.

I'm a little confused why he cares about nofollow links--if Google doesn't "own" the internet, what does nofollow matter? It's certainly better than not having a link at all.

Furthermore, the "credit" he so desperately wants publishers to receive is only a thing because of Google Page Rank in the first place.


Furthermore, the "credit" he so desperately wants publishers to receive is only a thing because of Google Page Rank in the first place.

That makes no sense. Citing an original source, such as an article that inspired your own work or with relevant background for ideas you're building on, is common courtesy in any field, and it's directly useful to your readers in the Web case.

Amazing as some people apparently find it, it is still quite possible to "surf" the web, following links from site to site without relying on a search engine like Google to find anything interesting or useful. The fact that Google appear to be promoting policies that could threaten to this alternative is rather the point here.


> That makes no sense. Citing an original source, such as an article that inspired your own work or with relevant background for ideas you're building on, is common courtesy in any field, and it's directly useful to your readers in the Web case.

The OP is against "nofollow" links which only affect search engines. The links work just fine in a browser and visitors won't know the difference. Publishers aren't scared to link, they're just wary of vouching for a site by not using nofollow.


There is this omnipotent master somewhere in the cloud. His power is large, but his ways are unknown. To please the Lord of cloud, strict rules have to be followed. Rules, that, when not followed, will cause the cloud Lord to strike you down with vengeance. Unfortunately, the cloud Lord is intentionally vague about these rules.

Some people have taken upon themselves the task to discover the rules that please the cloud Lord. In the process these priests have found a large number of arbitrary rules that have to be followed strictly. Breaking any of these rules is a reason for the cloud Lord to banish you to the dark corners of his empire - or so the priests say. For only a small fee the priests will give you a glimpse of the rule list that might you good standing with the cloud Lord. It may work, or it may not, because the cloud Lord works in mysterious ways.


The funny thing is that the Cloud Lord wants to set his rules so that if you do what's right and honest, what is valuable to the world, you will prosper, yet people constantly try to game the rules, so that the Lord's intentions cannot be fulfilled.


Do I understand correctly that Google will try to punish you (or the other party) if you host links in exchange of money but that Google is doing it in its search results ? (And I receive from time to time unsolicited regular paper mail from Google so that I advertise my enterprise through them.)

Page Rank is a neat idea, but making everyone listening to Google so that Page Rank remains meaningful is stupid. What do you do if suddenly Google thinks Twitter or Facebook are to be considered spammy link farms (which would be true) ? Do you ask everyon to delete their tweets linking to you ?


Google's ads don't flow PageRank and thus don't affect search engines.


Do they (along with the personalization aspects of the relevancy algorithms) boost aggregate awareness & CTR - which in turn further drive rank?

A look at Ask.com's history on a tool like SEMrush makes this line of thinking rather compelling.


This has happened to me too. A client who bought ads on my site wanted the links pulled so as not to make Google mad. That Google would think me a spammer is very very broken.


If you were selling links Google isn't that far off...


I'm selling complete ads, that certainly contain links, just like Google does. They are contained in posts clearly marked as sponsored posts. These are not link farms, spam, or payed links. Advertising should not be a crime.


You're in dangerous territory and are not selling "complete ads". You're selling links.

http://googlewebmastercentral.blogspot.com/2013/02/a-reminde...

> Please be wary if someone approaches you and wants to pay you for links or "advertorial" pages on your site that pass PageRank. Selling links (or entire advertorial pages with embedded links) that pass PageRank violates our quality guidelines, and Google does take action on such violations.


If you are selling "ads" that transfer PageRank, you are selling links.

I do not care that you do this, but you should be aware of what you are doing.


My understanding is selling links is an SEO scam. Are my "ads" a SEO scam or are they real companies to trying to sell real products to real people? How can you conflate the two?


Would your advertisers be fine with you adding nofollow? If so, do that and you're fine. If they aren't fine with nofollow you should realize that the other stuff is window dressing and you're really selling pretty links (which means watch out, Google is looking to penalize you).

Real companies selling real products to real people get in trouble for scuzzy SEO all the time. JC Penny comes to mind. Or RapGenius (though they aren't even selling anything!). It's the techniques used that are the issue, not what is being promoted.


Shouldn't we be talking content and not form?

Let's say a reader of mine is searching for a job and my site has a new great job at Apple. Apple advertises on my site because it attracts a certain kind of person. Shouldn't that person be shown my ad because it's both timely and highly relevant?

That the job is on my site is a valuable signal. I don't use an ad service, I place every single ad and I have done so since the start. I check every single link. I check every single ad. I personally talk with every single advertiser.

It's not hard for me to imagine an algorithm that does similar sorts of calculations. Oh, this link goes to apple. I've heard of them before. It's for a job. Scalability Engineer seems to be a valid occupation. Looks OK to me.

You are saying all such signals have no value and should be filtered by nofollow?

In actuality I've not given nofollow much thought. I guess I may now. I've lived by the idea of creating quality content, being honest, and doing a good job, thinking Google will figure it out. That unfortunately doesn't seem to be the case.

The irony in this age of targeting and personalization valuable signals are being lost.


Yes, none of that is problematic. You should just make sure you're not indiscriminately linking to third parties who pay for the privilege. In your case nofollow would be very simple and would prevent any damage.

Quality content is awesome and I'm sure your site is great, but Google's thought process here is that your site's reputation should only be handed on to sites that earn it organically (e.g. not through their buying it).

It's really for your own benefit, with the way PageRank works your outgoing links are votes that are as powerful as your page's reputation is good. If you link to bad sites your page's reputation takes a hit and thus your PageRank goes down. If I pay you to vouch for me (e.g. link to me) that only means that I have money. If you vouch for me without any money that's a much better signal that I have quality content.


It's not a "scam" per say.

Google has established these rules to prevent the commoditization of PageRank. They want links to represent an editorial choice and indicator of quality, not marketing budget.

There is nothing wrong with selling ads and visibility on your site to companies that wish to purchase this, but you should nofollow the links to prevent running afoul of Google's TOU.

Again, I don't care if you sell links on Fiverr and make a million dollars. I'm just explaining the rules to you.


you need to proxy paid links through another domain to make it clear that they're ads, or something.


So you're saying all ads are 'paid links', except I presume Adsense ads. That sounds fair and reasonable.


No, but if you're selling something that transfers PageRank you are (in the eyes of Google) selling links. It's that simple.

http://googlewebmastercentral.blogspot.com/2013/02/a-reminde...

> Selling links (or entire advertorial pages with embedded links) that pass PageRank violates our quality guidelines, and Google does take action on such violations.

It's an easy test: if the advertiser isn't OK with you not transferring PageRank you are selling a link and not advertising. It doesn't matter that it's good content or that it's not spammy (that just makes you the publisher feel OK about it), you're selling links.


This whole thing seems bizarre. Pagerank smells broken to me.

You can't buy links, but you can if you do it from us. What a scam.

If pagerank can't differentiate "blackhat" seo links from organic links it probably needs fixing.

In regards to this article its a bit strange. Reading the email its pretty obvious how it went down:

Company hires dodgy seo company who sets up a bunch of links

Google detects it, sends out warning listing all links to their site

Company mass emails every site owner with links pointing back at them

I don't think they were really asking the links to be removed from the authors site at all, it was aimed at someone completely different. All they had to go by was the referrer and the link.


> You can't buy links, but you can if you do it from us. What a scam.

You seem to be confusing how Google's advertising work. You can't buy a page rank passing link from Google. Adwords and Adsense ads do not pass PageRank. No legitimate advertising does (for starters because they all want to track visits and send you through redirects, but also because it's usually injected remotely and not part of the page that can send PageRank).


But that is my point what counts as legitimate advertising? Who decides whether a link is non passing advertising or not. Google is the gatekeeper here. They get to decide what links are "advertising" and what are naughty naughty black hat so. Are you saying that if the seo guys labeled their spam links as advertising it would be legit and google would just ignore the links?

Now they are telling you how you should advertise as wellas which adworks are allowed to sell you links. Want to try some viral marketing where you dont tell the user it was advertising - just brand awareness and a link back? Oops no that's trying to mess with Page rank were going to flag your site.

Its ludicrous.

Look I know they're not out to hit the good guys and the spammy tactics of bh seo peeps is gross as hell but I dont think you can really draw a definite line here without collateral damage being inflicted which is what I think this article was trying to point out.


nofollow makes it extraordinarily easy to stay out of trouble with any type of advertising you can think of. If your SEO guys used nofollow on their spam links it would not matter if they were paid for links or not (they wouldn't have a positive SEO effect though and would be a waste).


Applications are open for YC Summer 2023

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: