
Robots.txt - error54
http://explicitly.me/robots.txt
======
dkhenry
Keep up the good work Google. I hope this serves as encouragement for them to
continue to screw over those who want to turn searching for relevant
information into a war for who can cheat the system the best.

~~~
CKKim
Totally with you. I was enjoying the amusing tone of the piece but he lost my
sympathy at:

"Before, SEO was the game of really skilled people, you know like professional
poker players. You had to know the game, know when to raise, and when to
bluff."

Then with the "Maybe I should find another job?" I thought this has got to be
satire. Well played, sir.

The way the piece is framed, the writing style, surely a work of humour?

~~~
illuminate
From his twitter postings, I don't believe this is fake.

~~~
CKKim
Ah, thanks for this and your comment elsewhere in the thread. That Twitter
link, for those also intrigued: <https://twitter.com/rishil>.

------
buster
Am i the only one who thinks that SEO is more a plague then it is a real
profession? Google making it hard for SEO is exactly the right thing to do.
Making it impossible for SEO should be the ultimate _goal_ of a good search
engine.

~~~
homosaur
Depends what you mean.

If you mean people whose goal it is to help site owners learn how to structure
their data intelligently for dissection by machines, people who make companies
think very hard about their content, people who encourage high-quality
crosslinks and meta tags, and people who generally help sites manage data,
then no, SEOs are great. They help users and companies alike.

If you mean the dirtball black hats, then yes, they are a parasite that
deserves no mercy and should be eliminated at all costs.

~~~
chc
<http://en.wikipedia.org/wiki/If-by-whiskey> seems relevant.

~~~
buster
Yes, SEO has clearly a negative notion for me. To deceive the search engine
and make it less useful for me is practically the job description of SEO, in
my opinion.

For all the "no-evil" stuff, it's just plainly the work of a website
programmer. Properly formatting your HTML, making it accessible and easily
"readable" for machines is part of your job when you create a website. Same
for creating a robots.txt. If you say many people don't know how to do that,
then those people are just bad in creating websites and they need help of a
professional website programmer/creator. No need for a term like SEO here.

~~~
bones6
I can see what you are saying but I see SEO a bit differently than being
simply good web programming. It's about content organization as well as
information flow. Some of the time that comes naturally to a webmaster, but
there are so many content rich sites with poor web programming and poor
organization. SEO is a niche side of web programming that fixes that specific
problem. The skills of an SEO sometimes don't even cross the mind of a "good"
web programmer because of the unknown unknowns factor. They just haven't
thought about even attempting to rank a website on Google. They aren't
investigating the competition to see where they fit, and how to be better.

------
bengillies
It seems that most people here would like it very much if SEO died a horrible
death as quickly as possible. Let me try and paint a (probably somewhat
sketchy) picture of why it should exist (I have nothing to do with SEO
consulting btw):

1\. You have a business. 2\. Said business has a website. 3\. Said website is
bringing in no discernable revenue for your business. 4\. Poking around for
reasons, you discover that your website ranks really lowly on Google. 5\. You
have no idea why. 6\. You pay an SEO consultant. They take a look at your web
site, and your overall web presence, to try and find out what's going wrong.
7\. They discover that your markup is terrible, the wrong things are enclosed
in header tags, etc. Also, you have little to no visibility outside of your
website. 8\. They advise some sensible changes to your markup, and force you
to start posting things on Google+, Twitter, Facebook, etc. 9\. Your ranking
improves. As does your revenue.

Which bit of that process is evil and wrong?

Of course, there will always be a shady side to SEO (people taking it too far,
etc), but there's a valid reason reason it exists: many people simply do not
know enough to get the basic components right. SEO consultants help them do
that, and improve revenue as a result.

~~~
cs02rm0
I think the general contention is that the process you describe is just using
someone who's going to write sensible HTML not SEO. The shady side of actually
modifying code not for easy-to-read, browser-efficiency, etc. is SEO.

~~~
RBerenguel
I have to disagree. I consider part of my job to be some SEO (not my primary
task and neither the most important, but I take care of it at my company) and
its definitely the first part of what you sketch. And trying to clean all the
second part stupid stuff someone else may have done before.

I don't see it as a game against Google... I try to make our sites as useful
for humans as possible, and hope Google agrees with a decent enough ranking so
humans can visit it and enjoy the content. But when it doesn't, then the game
is between me and the "other SEOs" who may not see the point of making
something useful for readers, not machines.

~~~
waxjar
Writing sensible HTML is optimising for machines. The Google algorithm runs on
machines, so they'll like it and benefit from it. Browsers, screen-reader,
future colleagues will also like it and benefit from it. Calling this search
engine optimisation seems a bit weird, though it'll certainly help Google
process your website properly.

If you read the OP, his mean concern is the tiny subtle changes in the
algorithm that drive him mad. He calls optimising for them SEO. It is
considered shady and evil by some, myself included. (Ultimately it's about the
content, not about who best reverse-engineered Google's algorithm.)

------
hornbaker
An SEO consultant complaining about Google's frequent algo changes is like a
mechanic complaining that cars break down too often.

~~~
tlrobinson
It's more like new cars being introduced without service manuals, forcing the
mechanic to continually figure out how to fix them.

~~~
mixedbit
I don't think this is a good analogy. Google's job is to make sure good
content is ranked high and crap is ranked low. You should just create content
that is useful to others and let Google figure out how to discover it and rank
it high (they are good at it).

To create useful content, you don't need to know Google internals.

~~~
dohertyjf
And we would _just_ do this if Google actually made good on their promises to
do this, but I see an awful lot of crap still ranking in the SERPs especially
because of hacked sites and link networks.

Full disclosure - I work in SEO.

~~~
illuminate
The solution to no-content black hattery is not to promote your no-content
site with black hat techniques. Nobody here is criticizing site tuning and at
the same time defending content farms. Why defend misbehavior as a
justification for misbehavior?

Note that I am not saying that you yourself perform any of this in your role.
Our anger is not at site tuning and placement but the techniques used to do
so, which you appear to be unnecessarily defending as essential for any site's
survival.

~~~
dohertyjf
Your comment doesn't make any sense. Of course the solution to something bad
isn't promoting it with something bad. That's just ridiculous and I would
never argue that.

You're automatically assuming that SEO is bad, or is all blackhat, or is just
about promoting crap. I would argue vehemently that good SEOs only want to
promote good content. I turn down many freelance gigs because the company is
shit.

I think you're confusing legitimate SEO with crap affiliate marketing. Big
difference there, buddy.

~~~
illuminate
You replied a person who said to just create content that was useful to others
with suggesting that you would do that, only if there weren't hacked sites
jumbling up the rankings. It ~seemed~ to suggest that you were less interested
in original, interesting content than you were competing with the top sites
regardless of how either of you got to that spot.

~~~
dohertyjf
I said that we'd _just_ do that if... The _just_ implies that we do that plus
other things.

------
mixedbit
SEO has made a lot of damage to the Internet. 10 years ago links really
expressed people preferences, and it was easy to get linked and ranked high
when you've published useful stuff.

Today, almost all large sites that people use to share links to original
content use nofollow links. Because of this, it is common that when you Google
for some original content, you will find links to aggregators (Stack Overflow,
Reddit, HN, etc.) but the original page will be very far on the result list.

~~~
handzhiev
Well, actually Google did this first, because they started using links as
"votes". Then everyone started hunting for links instead of trying build do
good content.

------
artursapek
I thought he was gonna ask for a job.

~~~
gcatalfamo
me too. in that case it would have been cool.

~~~
explicitlyme
Ha! I should have... Maybe in the next version...

------
tibbon
I still strongly feel that the best way to achieve excellent SEO is through
building human-accessible pages. How is your site on a screen reader? If its
easy to navigate by braille keyboard or voice output, then it probably is well
enough that Google should be able to make sense of it.

Better yet, make it so that your CODE is readable and understandable easily by
someone who doesn't have perfect vision.

If you achieve this and create great, relevant content, then Google will snap
it up and the customers will appear. No other tricks needed.

~~~
explicitlyme
10 people targeting the same keyword. Only 10 spots on page one of the SERPS,
if the on site content and site is well built, then who gets the top spot?

The machine isn't clever enough yet to get the perfect answer, and till it is,
humans and marketers will continue the manipulation of visibility in pursuit
of traffic and revenue. It's sad, but it's a fact.

------
moccajoghurt
I wonder how this post got to the top even though the sentiment in the
comments is that this posts sucks.

~~~
michaelhoffman
We can't downvote posts.

~~~
Terretta
But we can flag them.

~~~
michaelhoffman
Yes, but why would we?

~~~
illuminate
Right, I'd only do so if it was an outright scam.

------
woohoo
Nobody else read this as a joke? I read it as a joke (and a slightly racist
one at that).

~~~
CKKim
I read it as a joke by the end but, worryingly, didn't automatically
recontextualise it as racist in light of that. Which, I think I agree with
you, it is, if it's a joke and all the East Africa stuff is made up for
"comedy value".

~~~
nirvanatikku
....? where did the idea of "The East Africa stuff" being made up come from?

~~~
CKKim
Nowhere. It was speculation on my part extrapolated from the premise that the
piece was written as a joke and nothing in it could be taken at face value. A
little research on the guy suggests I was reading too much into it. Here he is
<http://www.linkedin.com/in/rishilakhani> and his languages are listed as
"Hindi, Swahili, Gujarati, Punjabi" - we don't need much more than that to
connect the dots.

------
webwanderings
I think SEO is practically dead, and for good. The Social Web is supposedly
lot more democratic and apparently a better choice.

~~~
illuminate
"I think SEO is practically dead, and for good. The Social Web is supposedly
lot more democratic and apparently a better choice."

Subverting "The Social Web" occurs to feed users to their advertising or other
attempts at monetization, they don't get paid to Tweet.

~~~
webwanderings
Well, there's a negative as well, hence I used keywords like supposedly,
apparently etc. But wouldn't you rather your insurance-selling uncle visit you
for yourself, instead of an intent to sell you the insurance? I think those
days will be here eventually.

~~~
illuminate
Say people eventually buy everything or get some form of advertising kickbacks
from "the Social", do you think that there won't be ways to game those income
streams?

------
napolux
That robots.txt is so big that even with comments maybe google will drop it
after trying to parse it for a while....

~~~
zalew
robots this size aren't that unusual <http://www.bbc.co.uk/robots.txt>

~~~
chayesfss
wow, just realized this is a great way to find information about websites

------
daigoba66
I always thought SEO was supposed to be a cat/mouse game. The search engine
tries to balance the search results fairly. The SEO consultant tries to game
the system and get certain links to appear ranked higher, whether fair or not.
Isn't that just how this works?

~~~
perlgeek
gaming the system is blackhat SEO. There's also supposed to be the SEO
business where you actually improve the site's quality, indexability and
speed.

------
Igalze
Very cool stuff but I have to disagree with the general idea.

I've been doing SEO for 8 years and I have to say that I came for the fun and
money but I stayed for the challenge and algo changes.

I remember how, 8 years ago, it was all about "5% density" and link stuffing.
Today is content quality, social signals, link profile (not just quantity),
contextual co-relations and so much much more...

I think that I only truly fallen in love with SEO when Google started using
rel=canonical. This is when it really became a game of poker and not a "hungy
hungy hippos" with links in it.

------
daGrevis
You should use `humans.txt`. :)

<http://humanstxt.org/>

------
phillgeorge
Bmac27, what kind of enterprise seo toolset do you recommend ? I am currently
evaluating Skycara, Brightedge and SEOcatalyst but wanted something else to
try and look at. I'm runnign a site with 15,000 pages, 10,000 errors and about
20k keywords

