Compare this to regular websites. Lots of smaller sites will end up paying a "whitehat SEO" firm to work on a "link strategy". These firms will claim up down and sideways that what they do is legal, ethical and follows Google's rules. But, what they actually do is either (1) create networks of fake sites to provide backlinks on certain terms to artificially boost the site, (2) place spam comments using bots on legitimate sites to do the same [not that this will thankfully no longer work well due to the latest Google algorithm update], or (3) pay legit sites to place backlinks to artificially transfer pagerank the same way that RapGenius did. Now, these other sites, when they get caught, they get a manual action or a smackdown. The difference? They have to actually pay the penalty. Arguing that they didn't know usually doesn't work. The penalty is LONG. They can't call on their VC firm to make calls at Google to give them a get out of jail free card.
I'd like to call on Google to create a public policy on how they handle these manual actions with some clearly defined penalties (example: 3 month manual action of 6 PR drop, etc) and to consistently enforce them across the board. That way a mom and pop site that pays a 'whitehat SEO firm' and gets caught doesn't have a worse time than a site like RapGenius that purposely engages in blackhat SEO who can use their VC connections to get out of having to pay a penalty in under 2 weeks. It's also sad that Google gave this get out of jail free card to a site whose entire business model is based around other people's copyrighted works which RapGenius doesn't have a license for, doesn't pay for, and publishes illegally.
As a search engine user, I don't care whether the search engine firm is "fair", whether their rules apply equally to everyone or not. It's not a country, it's a search engine.
You can say what you want about Rap Genius, but they're just about the only lyrics site that doesn't suck. As a searcher, I want to get to sites that don't suck. In the case of lyrics, that's Rap Genius.
Google has an interest to show me the best results. How they determine those results, and how much of that process is manual, is up to them. It's nice that they decided that penalties and virtual spankings are a good thing in the long run, and it's lovely that you think that they're too inconsistent and unfair about it, but I'm a user. I don't care. I just want the best results.
Heck, sometimes I have to click more to get what I'm looking for with RG. After trying a few things, I was able to reproduce this just now with a search for "forgot about dre lyrics" (no quotes). This is the second result I got: something google shows as "rapgenius.com/...dre-forgot-about-dre/Eazy-es-ice-cubes-and-docs-the-s..." that does a redirect to http://rapgenius.com/Dr-dre-forgot-about-dre-lyrics#note-252... which pops up an annotation over the lyrics. That's not what I was looking for.
All that to say: it's not clear to me that RG really sucks less for "[song name] lyrics" queries.
I think Rap Genius is great for "song meaning" queries, but their interface is just as distracting (if not more) than other lyric sites that "suck."
I've done a few spot checks on Poetry Genius, at least, and the annotations have consistently been either wrong or irrelevant, even for works that are well documented (e.g. The Waste Land).
My guess would be that if Rap Genius users aren't getting it right when large bodies of secondary literature are available, they can't be doing very well with pop music lyrics either. I could certainly be wrong, though.
The whole point of Google's search algorithms is to "curate" results using algorithms and metrics based on the behavior of users on the Web. In a sense, the curation of content is crowdsourced, drawn from the activity of the entire network.
When Google takes actions that inject nepotism into this system, it sets a dangerous precedent. I don't want individual people at Google arbitrarily prioritizing what content I should be seeing. (There are other sites for that kind of curation.) I want the collective actions of Internet users to decide this via Google algorithms. There's an important difference.
A lot of people are quite apathetic until the impact is direct. They recognize their own usage of the service rather than empathizing with a story from someone who was soaked. Here is a perfect example from a smart, tech savvy person who is big on self-interest / own vertical, but entirely apathetic toward other verticals - and this post is less than 4 hours old
The tricky part in all of this is that one man's life work is another man's spam. Everything at scale has some mix of original editorial vs user generated content vs scraped & reformatted content vs policing of the mix. Ultimately the entity with a built in audience will be able to win more and more verticals over time due to preferential placement. A lot of people are unaware of how the knowledge graph will keep extending to swallow a larger and larger portion of the overall query stream. Anything of significant value that can be structured eventually will.
One of the more alarming articles I have ever read was one from 2007 where the New York Times explained about Query Deserves Freshness coming about after Google found that their own new finance service didn't rank as well as it should have. That was the beginning of the end for the indy web.
The 2009 recession was the icing on the cake that cut Google's revenue growth rate & got them fired up on moving beyond direct marketing to focus more on brand, brand, brand.
So, while Rap Genius is a cool website, I wouldn't say that all the other lyrics websites suck.
With the goal of minimizing overall dirty SEO, Google shouldn't get caught up in an over-emphasis on punishment. Especially when the website being punished is actually the website that should be chosen, and they only use SEO to compete with other SEO.
It's not a black/white issue, sure for a very spammy website, it would work less (because that website is still out of the picture). But for a grey website like rapgenius, it would surely suck a lot more, because they would lose all their brand image, etc, which is quite something to build.
Google wants its results to be the most relevant, not contain the best cheaters. Allowing cheating would set a bad precedent eventually leading to deterioration of search quality.
I'm confident in this assertion because people have, at various times, claimed that other search engines provide them more relevant results, yet for me, those search engines almost always failed to even match Google, and never exceeded it.
This was also the case pre-Google -- e.g. lots of people swore by AltaVista, but I almost never used it, because it never seemed to work well for me. I actually remained primarily a WebCrawler user for quite a while, right up until settling into Google in the 1999~2000 timeframe.
I don't want to be limited to search engines that provide what some bureaucrat has decided are the most relevant and/or neutral results. I want to be able to choose the search engine that provides the most relevant results for me, and I have seen no evidence that what's relevant for one person is necessarily relevant for everyone else.
That is precisely my point.
(And if you are looking for personalisation, an algorithm isn't necessarily counter to that but a global smack-ban on behaviours most certainly is.)
Any external notion of correctness imposed industry-wide will destroy that choice.
On the one hand that could be criticised as an illusion of choice in the market (behind the curtain there is often ownership across entire industry sectors, regardless of 'competitors' within a sector). On the other, governments quite correctly get a say in any case (what do we representatively govern otherwise - the alternative is to cede governance to global corporations). Google is particularly sensitive to the latter as it has a huge de facto monopoly and has become part of our infrastructure - it is absolutely a huge target for regulation wherever it has traction.
Talking of competition, is there anything viable in the shape of an open source effort where the algorithms and indexes could be crowd-managed - perhaps indexing via browser plugins?
I think a comment I wrote a couple months ago about Facebook is relevant here.
Infrastructure is what services are built on. It's a prerequisite, not the end result, and its absence is extremely costly.
Google is not infrastructure. Like Facebook, it could disappear tomorrow, and we'd all just switch to other search engines. I assume Bing would immediately pick up most of the users, either directly or via Yahoo. I wouldn't like it as much, but I would not be materially harmed.
Google has done nothing to make it harder for someone else to build a search engine except raise the bar for quality. It has a better product for most people, so it has the most users. That's how it's supposed to work.
Using Google's popularity as an excuse for regulation is nothing but a demand for mediocrity. It's saying good things aren't allowed to exist, and that people shouldn't have the right to choose (or make) better products.
It's saying "CONFORM!".
 Well, Google does offer something closer to actual infrastructure of course, in the form of their App Engine/Compute Engine offerings, but they're even less dominant in that field.
Do you have any idea how evil that sounds? If that is the argument, this is my answer: I will take up arms and die fighting to prevent the realization of such a goal.
Media ownership limits are justified only because of a resource scarcity. They do not apply to Internet publications, and with search engines, all I have to do is type in a different domain name. There's no spectrum to be monopolized.
Media ownership limits are about market monopolies, not scarcity, newspapers being a prime example.
If it undermines a country's judiciary for information about crimes to be published, that country's judiciary needs to be undermined. Once information has leaked beyond law enforcement officials, there is not and should not be anything anyone can do to prevent its spread.
I don't care what mission Google is on. Rights are not dependent on motive.
Newspaper ownership limits exist only in relation to broadcast station ownership. There are no separate limits on newspaper ownership.
Remember, we're talking about someone (Matt Cutts) who penalized Google's own Chome website for improper links. If there was ever a conflict of interest, that would be it.
The idea that Matt was doing RG a special favor through a16z or other investors is an extraordinary claim, and we're going to need more evidence than just gut feeling.
Also, I have invested in Perfect Third (the company that
makes the WakeMate), Zencoder, Cardpool, Tasty Labs,
Drchrono, Grubwithus, PoundPay, Apportable, Mailgun, and
Parse. I have also invested in CircuitHub, PlanGrid,
Pixelapse, ZenPayroll, Trigger.io, Zenefits, and True
I have also invested in Lowercase Capital (Lowercase
Ventures Fund I), Y Combinator (Y Combinator Fund II),
Lowercase 140, and Lowercase Spur.
That's not any evidence that RG specifically had a direct line to Cutts, and that this actually influenced the outcome. Again, we're talking about someone who penalized Chrome. I don't know of any greater conflict of interest than your employer.
In an argument about someone's motivations, I am inclined to give them the benefit of the doubt in the context of their past behavior.
The incredibly fast timeline is pretty clear evidence of communication considering the standard response time is never and the standard unpunishing time is never and the standard result is basically "change your domain".
They are back because they did a lot of work finding all these bad links and using Google’s disavowal tool to kill them, not because of their VC funding or whatever other influence you think they have on Google.
Removing 200 links out of nearly 200,000 links in a holiday week is not a lot of work. In fact, Google has told others waiting to be reconsidered that the wait time is multiple weeks.
Further, some sites which submitted reconsideration requests before RapGenius was penalized have not received a response. And in many such cases those other sites:
had far fewer links, removed a far greater quantity of links, disavowed a far greater quantity of links & were involved in something that was nowhere near as overt/intentional on the behalf of the webmaster (I've seen cases where the notices came after a payday loan hacker mixed in someone else's sites with their own links to create cover for their work)
They got 200 webmasters to remove their links, a few didn't, and the rest ended up being "site scrapers" who steal other people's sites. They just added those to disavow. Seems like they acted very quickly to rectify every suspicious link they could find.
But yes, they got special treatment in having the manual penalty removed after only a few days. But it's a legitimately popular site. You'd expect Google to do that since its harming Google's own users during the penalty.
There is no such thing as fairness. It's a paradigm of the feeble minded. Most likely a defense mechanism - "look guys, he cheated on the exam, let's tell the teacher!".
If no, please explain.
If yes, an important follow up on that front: how dystopian is it that the company in charge of organizing information requires spreading misinformation to succeed?
So on many levels, keeping it vague and saying simply, in so many words "just be good, don't do anything abusive" like many services do for high-volume APIs (e.g. YouTube, last I checked) -- makes for a pretty good, common-sense fit in this case.
It also puts the emphasis were it should be (in my view): not on "don't do X or Y, OR ELSE" but on "look, just be ethical, and use your head, please."
It's just too bad for RG that they didn't head Google gentle and clearly (enough) worded advice, first time around.
I agree with you in theory, but in practice it sounds like you expect google to act as if they care about things like fairness. They are a company. The bottom line is what matters. If you think it's unfair, then try to hit the bottom line by using other services and convincing your friends and family to do the same
RG is in a commodity industry. Lyrics are lyrics are lyrics no matter where you get them.
Yes, RG is better than most other providers. However, I suspect that the number of Google users that actually care about their differentiation (like annotations) is vanishingly small.
And even if, say, in Microsoft scenario they might benefited from using they monopoly position more than were penalized (can't for sure claim, because I don't know real numbers), it doesn't mean that companies should't not care about fairness. At the end of the day that is what competition suppose to do - force companies to think about fairness towards their customers.
So even if Rap Genius is not bringing revenue to Google, Google certainly can derive value from Rap Genius that contributes to revenue. However, since the effects are indirect, it's not easy (though not impossible) to put a monetary value on this relationship.
And if implicit or explicit risk protection (and back channeled information about potential upcoming risks) comes as a fringe benefit of direct investment, doesn't that "free insurance" allow Google Ventures to guarantee themselves more favorable investing terms?
Which does not happen when they ban your herb garden equipment website.
According to Compete.com about 5% of their searches were brand-related queries
SimilarWeb also shows most their top queries were unbranded
Alexa puts the branded-traffic number closer to 2%
(All these tools have errors, but sites like Amazon.com or eBay are more like 15% to 25%+ on the branded traffic front http://www.alexa.com/siteinfo/amazon.com http://www.alexa.com/siteinfo/ebay.com ).
As a user, yes I would like this site to rank up just so RG fans are saying they like RG result in google. But it is so unfair Google sends cryptic messages on webmaster for reconsideration requests but can entertain a direct call and lift off penalty. The least Google could do is to be transparent (fairness is hard)
But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly.
Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail.
Could it be that Rap Genius was special because it has Andreessen Horowitz?
Sure seems that way.
I am not going to stop using Google, but I am inclined to use them lesser now.
I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!
> They're a startup, and this was a problem which was either going to kill them or be surmounted quickly. Naturally, they surmounted it quickly.
Google laid out a clear plan of action for them. Is it hard to believe they finished it quickly?
They did not quite finish their action plan. They got away on reasonable doubt.
This is the first time I am seeing this defense considered by Google.
I willing to bet any other non-vc funded bootstrapped start-up would never be afforded this consideration by Google.
This rant is not a judgement on Rap Genius, I am actually glad they maneuvered around this.
I am judging Google.
They have killed other start-ups for reasons such as other sites farming and duplication their content more vehemently. Reasonable doubt has never been considered for anybody else and most likely will never be hence forth.
What about that gmail fiasco a few months ago? Users got blocked out of G services for not providing their real names and had no recourse to get their data for several weeks/months.
Just goes to show that in several cases it's not what you know but who you know.
In this case, the user experience for a Google user is worse when Rap Genius is removed. Google has some incentive to be more active in fixing the problem there, especially since Rap Genius was cooperative, technically competent, and aside from their "rapper swag persona", genuinely nice guys.
I run across Rap Genius now and then and have always considered it to be spam, just like yahoo answers, ehow, and mahalo pages.
The main driving force behind it is that anyone can edit the lyrics, and even better, anyone can ANNOTATE the lyrics to explain what certain sections mean.
Let's take for example the lyrics for Mumford & Sons' "The Cave": http://rock.rapgenius.com/Mumford-and-sons-the-cave-lyrics
I looked up this song one day trying to understand more of the allusions made in the song; I know the title itself alludes to Plato's 'The Cave' and wanted to delve deeper. Because of this site, I can see that Verse 3:
> So come out of your cave walking on your hands
> And see the world hanging upside down
> You can understand dependence
> When you know the maker's hand
is an allusion to St. Francis.
Most of the time when Rap Genius ISN'T the top result for lyrics when I google, I facepalm and go directly to their site.
I should emphasize here that this overly positive post isn't intended to condone their shady, spammy blackhat SEO tactics, but damnit if they aren't the BEST lyrics site by a fair margin.
Maybe I looked at the wrong lyrics, but I saw precious few explanations, instead most lines had jokes and/or totally unrelated pictures attached to them. Often enough the jokes is to simply restate what was said in rapper slang in plain english, which is funny for exactly the first time you read it, then it's just spam.
They have had some legal troubles, but are trying to sign licensing deals http://www.nytimes.com/2013/11/15/business/media/rap-genius-...
However, if their licensing deals are anything like what other sites have to pay for lyrics, it's going to significantly (entirely?) cut into their profit margin.
Is anyone else getting more and more suspicious of the scorn that people have been gratuitously heaping on RG? Like, did they run over everyone's dog?
Why? You've no doubt read 10x reasons people here have said so (in the other threads). Do you just want to argue? If you don't agree, fine, but clearly a large and vocal group does. So what? Move along - find other things to care about.
Rap Genius [...] and have always considered it to be spam
That's exactly how I feel too. Though the incentive for google is very unclear, the lack of clarity is explored further below.
> You have the causality wrong. Rap Genius has Andreessen Horowitz because Rap Genius is special.
Not saying Rap Genius is not a special lyrics start-up. Not even saying that it would not have been a good candidate for any VC financing them.
Though, Definitely saying that the special case leniency shown by Google sure make it seem like having Andreessen Horowitz is the only thing different between Rap Genius and scores of sites before with this and similar situations.
> In this case, the user experience for a Google user is worse when Rap Genius is removed.
And its not when other good sources are removed because they were link bombed or their content was scraped more than their competitors?
> Rap Genius was cooperative
Find me one person in their situation who would not be. It was to either co-operate or die.
> Rap Genius was technically competent
> Aside from their "rapper swag persona", genuinely nice guys.
I have nothing against Rap Genius founders, nor their "rapper swag persona" and I'll believe you when you say the are nice guys.
Aren't all affiliate programmes technically bad for the publisher after Penguin?
If the folks at Google can figure out how to detect this sort of gaming more reliably, that may end up in a future algorithm update, and no amount of pleading will matter for anyone affected by it - they'll have to fix the problem or continue to suffer the consequences.
Well there wasn't an algorithm update, there was a manual action. Google reversed the manual action after RapGenius stopped breaking the rules and cleaned up its mess. If this happened to you there would be a note about it in Webmaster Tools and you would be able to ask Google to take another look. I'm sure RapGenius had all hands on deck making sure they were up to snuff.
That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening.
Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position.
Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).
Should their investors not have made calls? However, it should be noted that they did their part.
One of my favorite quotes is this: "To save a drowning man, he must first give you his hand"
If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that.
BTW, it also helped that people really found Rap Genius useful. I'm sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad.
Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to.
BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment.
I'm sure Mahbod (my favorite) and co have learned that you do not go 'daaawging' when you are in a hole. I wish them success.
If Google applied these rules consistently with all websites, I would have no problem with it, but they don't. This breaks the myth that Silicon Valley is a meritocracy - as it shows that the connections you have are as important as raw merit, if not more
I have helped over 40 websites recover from Google penalty in the last 7-8 months. 80% of the websites seek help only after they have screwed up atleast one reconsideration request without taking the necessary steps.
Google typically responds to penalty reconsideration requests within a week, often less than a week. So if you have taken the right action and have submitted a really solid reconsideration request, its totally possible to get out of a penalty in under 2 weeks. Also, the backlink count was under 200K which is a reasonably manageable.
Rap Genius could have saved some time using tools like scrapebox to perform some of the scraping activities. But thats another story.
This is the PERFECT time for anyone who has done penance and is still being blacklisted by Google to document it all and go to the press.
There are A LOT of people/journalists waiting to jump on a story that shows Google has double standards.
But first they must be willing to do their part by documenting these facts and reaching out the journos. It will not come to them.
Like I quoted "to save a drowning man.....
Fairness does not exist in real life. Screaming "unfair" will not help you achieve anything. It's such a ridiculous situation for me. All these guys are sitting on the line, raging at real life because popular culture brain washed them.
I can't believe I'm saying this, but
What a bunch of stupid nerds!
Just as a thought experiment ... how tempting do you think it would be for Google to do the following:
1) Run a query to identify all companies that cut their AdWords spending over 50% over the last year.
2) Look at which of these companies are now getting significant numbers of click from organic results.
3) Find out what is causing them to rank high organically.
4) Penalize them using a generic message and refuse on principle to answer any inquiries as to why.
Apart from (anec-)data that seems to be pointing towards this already happening; should we be OK as a market with this approach? Especially when considering that the rules are so vague that almost every site is guaranteed to break some of Google's guidelines?
Please provide this evidence. I am getting a bit tired of this misinformation. Organic search and paid search are silo'd. To say that Google favors advertisers in the organic results or the other way round, is simply not true. It would destroy Google's credibility.
>how tempting do you think it would be for Google
It approaches conspiracy thinking. I love that, but I think HN is not the place. You are basically accusing a company of a very evil act, without proper evidence. Do you realize that many Googlers frequent this site? It would be a shame if all they get to read are conspiracy theories and baseless accusations.
>Especially when considering that the rules are so vague that almost every site is guaranteed to break some
Hogwash. Their rules are very clear and succinct.
Maybe if you want to spam or manipulate, then these rules are vague. Maybe when you don't read the guidelines then they are vague. I bet that if you can point to a rule that almost every site breaks, without them knowing, then Google will adjust that rule to be more clear.
You can not please them all.
>Penalize them using a generic message and refuse on principle to answer any inquiries as to why.
You can certainly not please the spammers that were caught with their hands in the cookie jar and then take to the internet to say that Google is unfair, and that the big brands get away with anything.
If I think of a lot of things my users would search for and try to create valuable information for them concerning those queries... is that valuable content or are those doorway pages?
If I give when someone few months of my service for free when they write a nice article with a link to my site... is that good customer service or a link scheme?
If I create a press release on PR Web and link to my service with an appropriate keyword ... is that participating in a link scheme?
None of these cases are all that clear cut and I for one am not comfortable with Google being the judge, jury and executioner.
> It approaches conspiracy thinking.
That's not the point. The point is that Google is a massive company wielding massive power.
When it comes to fighting a penalization their position is that they don't need to defend their decisions, implying that they can be trusted not to abuse their position and hence deserve to have the final say, legally, over these decisions.
I don't buy that and think it's about time that contesting Google penalties in court becomes a regular thing.
Create a useful, information-rich site, and write pages
that clearly and accurately describe your content.
Think about the words users would type to find your pages,
and make sure that your site actually includes those words
Doorway pages are typically large sets of poor-quality
pages where each page is optimized for a specific keyword
Some examples of doorways include:
Having multiple domain names targeted at specific regions
or cities that funnel users to one page
Templated pages made solely for affiliate linking
Multiple pages on your site with similar content designed
to rank for specific queries like city or state names
Avoid tricks intended to improve search engine rankings.
A good rule of thumb is whether you'd feel comfortable
explaining what you've done to a website that competes
with you, or to a Google employee. Another useful test is
to ask, "Does this help my users? Would I do this if
search engines didn't exist?"
Any links intended to manipulate PageRank or a site's
ranking in Google search results may be considered part of
a link scheme and a violation of Google’s Webmaster
Guidelines. This includes any behavior that manipulates
links to your site or outgoing links from your site.
Large-scale article marketing or guest posting campaigns
with keyword-rich anchor text links can negatively impact
a site's ranking in search results.
The best way to get other sites to create high-quality,
relevant links to yours is to create unique, relevant
content that can naturally gain popularity in the Internet
Effectively promoting your new content will lead to
faster discovery by those who are interested in the same
subject. Avoid: attempting to promote each new, small
piece of content you create; go for big, interesting
items. Avoid: involving your site in schemes where your
content is artificially promoted to the top of these
services. Avoid: spamming link requests out to all sites
related to your topic area. Avoid: purchasing links from
another site with the aim of getting PageRank instead of
Links with optimized anchor text in articles or press
releases distributed on other sites can negatively impact
a site's ranking in search results.
Note: I wouldn't expect links from press release web
sites to benefit your rankings, however.
So you can't figure out exactly how you can game the system... boo hoo.
According to who? Google? Is there any transparency or any way for an outsider to verify that?
Matt Cutts answers: Do AdWords customers get special treatment in organic search results?
If there is evidence that Google gives special treatment to Adwords customers in organic search then this could be verified as a lie. As long as this evidence remains nothing more than accusations and suspicions, then I choose to believe Google that there is no teacup orbiting Venus. It seems unlikely, but not impossible. I am willing to sway my view when I get evidence. Till then, I will treat it as a common newbie myth.
I like to believe that Google checked the link profile again and found a large amount of low-quality links removed. Google would not remove a penalty if the spammy links were still abundant. So that they are out of a manual penalty is justified.
Rap Genius showed good intentions to comply with the guidelines. They went out of their way to clean up their act. These were no doorway pages or elaborate linking schemes with their own servers. It was a Linking Bieber Scheme that is considered greyhat in the industry, and after this, will probably be less popular.
This was a large and visible PR drama. Of course both Rap Genius and Google were on top of things. Google never stated that this was to be a month-long ban. We don´t know how those thousands of other sites were penalized. Maybe the large majority is so spammy/crappy that they simply won´t rank near the top with their spammy links removed, on the merits of their content and audience alone. For sites that are entirely build on manipulating SERPs, then yeah, a Google penalty can seem permanent. To me, that is an ok thing.
About the connections. Sure they help. I don´t see what is wrong though with leveraging your connections when there are millions on the line. Maybe a connection humbly asked: Hey, we effed up, how can we restore trust? This thread makes it sound like the connections pressured Google into removing the penalty. That seems unlikely if we all give the benefit of the doubt.
"even I was scratching my head to figure out if this was actually a Google violation or not. Rap Genius’s apology post had the company deciding itself that maybe it violated guidelines that links should be “editorially placed." - Danny Sullivan.
At least now we know. This is blackhat.
RapGenious had no excuse.
Still, I'm glad they're out of penalty now.
I also view this post from RapGenius as a kind of a PSA.
They used the reasonable doubt argument, and won, they could have kept quiet about it. But, they let people know that others could appeal on reasonable doubt too.
Which is commendable!
Unfortunately, the underlying message some people miss when they feel this is about bashing RapGenius is that it is not.
The outrage is against Google. More specifically about 3-4 things:
1. Google has penalized sites for way lesser, and refused to reconsider, even when reasonable doubt was jarringly obvious (ex. some sites affected by panda/penguin algo updates targeted towards content/blog farms were penalized for as less as having similar posts type of intra-site linking as all content farms were using it)
2. Google will never apply this reasonable doubt argument for anyone in the future, unless they have friends like the investors at Andreessen Horowitz.
3. Google has given no indication on how they modified the algorithm for this special circumstance, the question that people are thinking is more in regards with if this was a team decision, or an individual's special favor.
4. It's a slap on the face for all white hat SEO professionals, webmasters and founders who have been burned by Google's no bullshit policy. Again its not about the policy existing, that's just a fact everyone has to work with. It's because now enforcement is clearly optional dependent on connections.
3] How do you know this was a special case? Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares), are you sure none of those get relisted as quickly as RapGenius?
I don't. I also don't know if this is not the case. Do you? This uncertainty IS the reason for the current loss of faith.
> Matt Cutts said they've relisted them quickly because AH slipped Google some cash?
I never said cash exchanged hands. I doubt it did. Can you with a 100% surety say that a special favor of any kind did not occur?
> Maybe RapGenius followed their cleanup policy correctly
What clean up policy. If there is such a policy why isn't it public, why doesn't anyone know of it? 
> (RapGenius was) relisted naturally
A manual modification of the algorithm is not what has been established as natural over the last decade+.
> Thousands of sites I'm sure re de/relisted every day that you don't hear about (because they aren't being talked about on HN and no one outside here cares)
Ah, but know tens of these sites that were penalized, as in know 'em in and out, know the whole team, know of every SEO practice they implemented, these only include those that were penalized for having site structures similar to content farms (themes, permalinks and similar post type cross linking between posts) that were not even replied too let alone getting a chance. I also know a lot of SEO professionals who don't know any site that was afforded this (I emailed 'em, if I hear of one I'll update the post).
These sites were all with 300K+ uniques a month && 1MM+ pageviews/month.
HN is not my only source of information and news.
> are you sure none of those get relisted as quickly as RapGenius
edit:  the only tool that I know Google allows webmasters to use to reduce potential penalty is the disavow links tool. In the past I have seen this only useful for people who were targeted by link bombing etc. at times it was because a SEO practioner did in fact use some shady link sources.
Kudos to RapGenius for talking about it in their blog post, nonetheless, the results were still too quick (not judging rapgenius or their connections... only targeting Google).
EDIT: added 
When did I do that?
I have not and am not suggesting that Matt Cutts or anyone at Google took a bribe of any kind whatsoever.
I have on the other hand said that there is some uncertainty if favors may have been given/used to expedite the process of re-listing.
Which if true, in itself looks really bad, simply because of the position of authority Google sits at.
Pray tell, when did a favor for a friend, acquaintance or anyone for that matter, regardless of their alleged wealth or social standing come to be defined as bribery/monetary value exchanging hands?
I used both interchangeably (see point #2), and I've edited for clarification.
If you want to keep going on the overly semantic line of argument, bribery = gifts, not just money.
Or quite literally if I go by a dictionary: an act of kindness beyond what is due or usual. Example: "I've come to ask you a favor". Synonyms: service, good turn, good deed, kindness, act of kindness, courtesy.
To further elaborate to address your failure to understand a semantic line of argument:
1) The terms Favor and Bribery are not interchangeable.
2) I have only used the word favor.
3) I have insinuated that there is possibility of an alleged favor having occurred in expediting the re-listing of RapGenius, and that this may have happened due to good connections from the VCs.
4) If true, this is not in any way illegal, by the law of the land (as opposed to bribery, which is illegal). And this would simply be a moral wrong doing to others who cannot use this speedy service based. This is due to the position of power Google has on web start-ups, websites and businesses.
5) In addressing your previous comment I stated that none of my statements can or should be construed as referring to bribes or any thing of monetary value, this includes gifts.
Meanings and Definitions:
 Favor - an act of kindness beyond what is due or usual
 Insinuate - suggest or hint (something bad or reprehensible) in an indirect and unpleasant way
 Possibility - a thing that may happen or be the case
 Allege - claim or assert that someone has done something illegal or wrong, typically without proof that this is the case
 Expedite - make (an action or process) happen sooner or be accomplished more quickly
 Construe - interpret (a word or action) in a particular way
 Bribe - a sum of money or other inducement offered or given in this way
Your insinuation is still baseless.
I suppose it's just a Hard Knock Life for companies that don't have prominent investors backing them. It's almost as if the investors pulled an American Gangster type situation on Google.
Well I suppose at the end of the day this is a reminder to Blackhat SEO users to Watch The Throne.
This is painfully true, and I know it has been discussed in prior discussions. Rap Genius is without a doubt THE BEST lyrics site available, hands down. Even if the other sites weren't ad-ridden scammy buggy shitty websites, if they got their acts together and actually TRIED, Rap Genius would still be top dog.
I don't care for the attitudes or personalities of the people in charge, and I am absolutely disgusted that they felt the need to scam their Google rank, but I am willing to see that some infractions can be rectified through penance, and am happy to know that they will not be forced to shut down because of this fiasco.
> "To save a drowning man, he must first give you his hand"
is very wrong from anything I ever read about the topic. Drowning leads to involuntary reactions that cannot be controlled, like trying to climb on the person trying to save you, endangering them as well. That's why approaching and grabbing a drowning person from behind is suggested, and maybe even knocking them out if need be. (unless you can throw them a lifesaver or something else they can grab safely, of course)
Er, what? You suggesting that Google execs are benchmarking the productivity of google searches? There are a million other "substitutes"...hence the shady SEO...
People tend to think in binary: A site is either spam or no spam. But spam is much more complex. A site can be 90% good and 10% bad. A site can turn spammy without their intention: comment spam, hacked sites. A site can pull out all the stops to hide their link buying. A site can be useful to have in index, regardless of some spammy behavior.
Avoiding the (questionable) debate on the productive capacity of planet-earth...the "model" of what customers' "really need" is questionable. Do they need (1) information that is factually correct?; and/or (2) do they need the actually meaning or interpretation of (1). And if its true a subset need (2) what is the ratio of (2/1)?
And only then does this framework make sense.
This is rapgenius. If it had to cross the line why on something so materially insignificant.
There were 31 thousand new backlinks added on December 31st (Holy shit, I just noticed that. WTF!). Another 1300 added on the 1st of January. It's normally 100 or 200 a day. Not sure how I'm supposed to manage the disavow process when I get that many new spammy inbound links a day.
Anyway, Franz Kafka would die again if he saw his text on this site (http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics), so let's just forget about this site and don't use it.
And about Franz Kafka, I think he'd turn in his grave if he heard about how petty people like yourself criticize a website that tries to make literary criticism of his work digestible and enjoyable for the masses.
It is a bad thing here: it shows that Google is apparently willing to act preferentially to favoured businesses, undermining the idea that search results are provided with neutrality.
Imagine a business in Germany, Vietnam, or even.. Texas having to do the same? It's pretty much impossible. This is what happens to everyone else out there.
Which is what the parent is referring to when he says that google results aren't as neutral as they claim to be. They clearly favored someone with leverage in this instance.
But this is the only instance that we know of. So far.
It's not a question of having to do the same, it just means that knowing a Google employee with some magic power is a valuable asset.
But yes, if you're a company that doesn't have this asset and you make a stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.
I still don't follow how that makes Google not neutral in any significant way.
>> stupid mistake like Rap Genius did your site might be off Google for a month or two, possibly destroying your business.
Sometimes businesses get wrongly blacklisted. Yes, it happens. They do get destroyed because of this, people do lose homes and jobs over this.
But this isn't because Google takes a month or two to sort this kind of thing out (a month or two is also not really acceptable - too long - given the impact Google has on a business). It's because experience shows that once you are on the wrong side of the big G, there's pretty much nothing you can do to get on the right side. It's no longer a question of a month or two, it's about registering a new domain, and starting from scratch. Sometimes you have to do this even when you didn't do anything wrong. This applies to almost every Google service whether it's Search, Adsense, Gmail, or anything else.
But Rap Genius definitely did something wrong. It was obvious for everyone to see. Yet, they managed to follow the 'recommendations' and get back in the game pretty quick. We know that this is probably because of connections because every business out there that gets blacklisted by Google will make sure that they are 'regulation-compliant' ASAP. Yet, many of them never manage to make it back and those that do take years at times.
Yes, knowing the US President helps, but that's provided you can get your work done without knowing him as well, in a reasonable time frame. Especially when it may make the difference between running a thriving business and being homeless.
QED: Monopolies are bad.
They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku?
They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google's fault and they should be reincluded? I don't get it.
RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want.
I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.
Are you arguing that what Heroku was doing wasn't wrong?
Comments on every single RapGenius post on HN have been filled with vitriol toward the founders, their attitude, and the culture they foster.
More shady SEO practices from Rap Genius, disappointed but not surprised.
The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics.
Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to.
The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages.
We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad.
For more info on the "Worse is Better" philosophy that caused the current jank situation to persist for so long, see this talk: https://www.youtube.com/watch?v=X45YY97FmL4
And some of those are not even poems:
This one is particularly weird / egregious:
You're lucky if you get a decent reply from customer support to begin with..
Your best bet is to change domains, fill in a redirect and try to build up backlinks again. It'll take a year or two and hopefully no competitor pops up in your space.
Or, you can try and get Andreesen-Horowitz interested in your business.
It entirely depends upon the scope of penalty. Unless you are completely deindexed from Google search results, you can always recover from a penalty.
Feel free to email me if you have any specific need.
<< Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>
- first, they use it with Google to show how sincere they were about "cleaning up"
- second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status".
Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected.
[Edit: clarified imagined comment]
From an IR perspective, search for lyrics is somewhat strange. I don't think it's easy to say if some hit is more relevant than the other. Sure, more popular sites may offer better UX (or simply less annoying ads), but at the end of the day it's a search for facts.
I wouldn't be surprised if google will serve those lyrics themselves some day. Freebase (and hence the google knowledge graph) already contains tons and tons of information on pratically any popular song (organized in a confusion way via recordings, canonical versions and whatnot) - artist, genre, album, release dates, length of a recording, etc. I don't think adding the lyrics as well is totally out of question. After all multible KBs are user curated and users provides lyrics.
Just like they serve "fact-hits" from their knowledge graph for queries like "movies by spielberg", google should then be able to serve the lyrics directly. While this is bad for lyrics sites, imho it is convenient for users and follows the same prinicple like "movies by spielberg" or "wife tom cruise"
We need more competition in this space. Sites need a search engine with real market share to compete against Google.
But what's surprising to me is that a company that knows how to automate things and work with data...why the f--- were they manually soliciting bloggers to post links? That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server.
It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google?
Or, was RG just doing it because they thought it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?
Moreover, this article feels pretty neutral/humble and techy, a far cry from the haughty attitude their founders usually display in interviews.
But now, Rap Genius co-founder Ilan Zechory tells us the site has returned to Google's search results. "It takes a couple days for Google to re-index everything, so search results are a little wonky right now, but we are officially reinstated," Zechory said in an email.
Somewhat ironically, Zechory served as a project manager at Google for two years prior to founding Rap Genius in 2009.
>> Wow that is troubling. So site that gets banned gets unbanned quickly and founder just happens to be ex google project manager
They're useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits.
Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?
Paying people to link back to your website on high-value keywords like 'justin beiber lyrics' from blog posts on other sites to artificially inflate your pagerank is 100% blackhat SEO. The payment in this case was a Twitter post about the link provider's website.
#1: it's a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice)
#2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it's not), they would have gone bankrupt ( i suppose) without Google.
#3: Without a quick response and/or measure to fix their shit on their end, investors wouldn't invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...)
BMW once got a penalty from Google, they didn't got easily off. But their business wasn't dependent on Google.
Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.
>Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text.
ie, you should really just become a member rather than relying on google...
Also, where was bing in all of this?!? ;)
There is no way they could get back so fast and so easily on google.
Google can't see a local business handing their customers/client a $25 gift certificate in-person for a random positive Google+ review. Along the same thread, Google would also not be able to see Rap Genius flying out a community outreach rep to meet with a prolific music blogger at a bar to go over an exchange for a positive mention and authoritative link. Just saying.
The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages.
What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks.
How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?
It's also rather weird how a lot of people seem to view Google as this public service for the people by the people etc, it's not. They can do whatever the hell they want and if you don't like it, simply stop using their service to send a message.
I just searched for justin Bieber lyrics and they are sitting in middle of page 3 on google.
They are ranking for branded searches, but not money keywords.
Although long term, I see this being a net positive for them due to the thousands of backlinks they got from all the press around this story.
I see it taking them several months to regain rankings...especially if the links they disavowed were artificially propping them up.
They make it a requirement of using the API that "nofollow" is not used: "Don’t ‘nofollow’ your links to Groupon."
This whole story made me fear that Google was choosing power over user experience.
And that should never be the case, no matter how offending the founders of RG might appear sometimes. Don't be evil!
Bought voters = manipulated SEO spam blog posts
Fair voters >> bought voters
Not reacting to the SEO manipulation would have meant accepting a fraction of bought voters.
Banning RG completely would have meant to ignore the even bigger fraction of valid voters.
Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters.
Fair voters always greater greater than bought voters. When you buy backlinks from the SEO consulting service, they gave you hundreds or thousands everyday, just like some people complained in the comments.
"Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters."
The problem is: how can they tell which is which?
I doubt this type of criteria is reliable. I collected some information from the user comments along with other research and wrote a blog post here:
Maybe you will also take a look at the possible solutions here: