
Do you debug your website? A small unimportant bug can have huge consequences. - swombat
http://kalzumeus.com/2008/11/17/do-you-debug-your-website/
======
patio11
I wish my blog would get picked up when it is NOT making me look like an
idiot. _sigh_

Edited to add: So I was trying for a less pithy comment when the phone rang
from overseas. Sorry about that.

Anyhow. Yep, this story really did happen to me. The good news: a one line
tweak to a little file I didn't touch in something like six months probably
added about 50% to my sales. The bad news: in those six months, this probably
had actual economic costs of several thousand dollars. Or, to put this in
relative terms, the worldwide economy just freaking collapsed and I _still_
managed to do more damage to my net worth with a freaking typo than the total
massacre in my IRA.

~~~
swombat
I don't think this makes you look like an idiot. I submitted this article from
your blog because it is insightful and shows how apparently small, unimportant
"display errors" can have very real consequences in terms of traffic and
revenue.

------
bemmu
This reminds me how much I wish it would be possible to attach notes on Google
Analytics graphs so I can look at them later and know what I did.

~~~
patio11
I know the feeling. That's actually a major reason why I keep the blog up --
it is probably my best single record of what the heck I was thinking when I
implemented something. I find explaining things to other people leaves better
hints than my cryptic "Of course I'll know what I meant in a year! After all,
I _wrote_ this comment!" documentation.

------
henning
Embarrassing story from someone I know: if you write your own billing
application, make sure you round numbers correctly, or you could cost yourself
hundreds of dollars a month in under-billing clients.

~~~
swombat
I'd say that's a nice problem to have.. if you're under-rounding by one cent
enough times to cost yourself hundreds of dollars, that would mean you're
making at least tens of thousands of dollars of sales...

~~~
henning
In the particular manifestation of this bug, he was under-charging by much
more than one cent. It was thousands of dollars a month, not tens of
thousands.

------
pchristensen
"Never send a commited Windows programmer to do a Unix sysadmin’s job."

Classic! (says the Windows programmer learning Unix commands and tools)

------
jyothi
Apart from being alert, cautious and the stress on testing, the takeaway for
me is

    
    
         "this bug made a live, growing website look largely dead to the world, and Google accordingly sent most of their searchers to get their bingo cards elsewhere."
    

Most web businesses need to understand how google search works than their own
businesses.

~~~
handelaar
This is the second time in a row that I've been unable to read a page on my
laptop because _you_ insist on posting quotes like that.

Cut that shit out, please.

~~~
jyothi
I don't see why I who commented on a valuable takeaway from the post should be
downmodded for this and your comment is upmodded. Yours is the one irrelevant
to the post.

PG, please add a official Don'ts in formatting. It is nothing but natural to
quote elsewhere on the web.

~~~
handelaar
Trying to be more specifically helpful, since I don't think I conveyed exactly
what the objection was the first time (or at least not so well that you
understood):

It's not that you quoted. It's that the quote is wrapped in <code> and <pre>
tags, and that breaks the layout for lots of people. I'm asking you not to do
_that_ , because its effect is obnoxious.

------
mseebach
Lesson 0: Don't do premature optimization.

~~~
tdavis
Lesson N: Use a system that doesn't require you to manually remove cache
files. Seriously.

~~~
patio11
So being that this is Rails there are a couple of ways I could have done this
(assuming, again, that I cached):

1) Cache the page in memcached, which would give me access to an easy time-
based expiry syntax.

2) Page cache. Instead of cleaning the cache with a one-line cron script,
write a 10 line cache sweeper. Then, install a plugin to periodically execute
the sweeper... or execute it with a cron script.

3) Page cache. Delete with a cronjob.

Why I didn't do #1: Memcached is overkill for my needs, and keeping another
process running on the VPS just gobbles its 256MB of RAM even faster, while
giving me another potential security/uptime headache.

Why I didn't do #2: Doesn't eliminate the dependency on cron, it just moves
the actual delete operation from a single line in the crontab to more
complicated Sweeper logic in the app. That is logic I'd then have to actually
maintain and whatnot. (And doing #2 wouldn't have helped me if I had executed
the Sweeper job as the same user I executed _rm -rf_ as, because it would
similarly have been unable to remove the file.)

I might be careless but I don't make choices totally at random. ;)

~~~
henning
"keeping another process running on the VPS just gobbles its 256MB of RAM even
faster"

Why not upgrade? On Slicehost, going from 256 to 512 MB is only $20 extra. If
you're running a business on it, I think that's a pretty reasonable expense,
especially if it can save you even a little time (within reason).

------
jrnkntl
It looks like all the 'my fail story'-blogpostings are quite elaborate for
what they're trying to say. With this story the same, it reads like a breeze
but all he says is: "Hey, I didn't delete the cache everyday because of a
fault I made in the crontab. That costed me pagerank and visitors." Doesn't
mean that I don't like reading those stories =)

------
gojomo
Little tweaks that speed page loading can also result in surprising increases
in traffic -- both visitors who stay longer and improved search-referrals. I
think fast-loading pages get a ranking boost from Google, too. (They're very
explicit that this is the case for AdWords landing pages.)

------
lallysingh
.. reading this after spending 2 days in emerg system fixing.

Only way this worked at all:

1\. Unit tests

2\. (Heavy) Load testing

3\. Version control, and knowing how to use it effectively.

What I wanted:

1\. Observability of what's going on in the system

If you have to attach a debugger or profiler, just to get a stack trace or
know the pressure on the GC, you're not in a good place.

------
sutro
Although the bug fix was only one line, it apparently necessitated a blog post
that was hundreds of lines long. Perhaps the author should ask Gavin Newsom to
produce a 7-hour series of videos about the bug fix?

~~~
eru
It's not the bug. It's the money.

------
josefresco
Only a real problem because of this: "one of the deployment steps had the side
effect of nuking the cache directory."

If that didn't happen he would have found the bug much earlier.

~~~
lpgauth
I doubt he released a new version everyday... It seems like this bug wasn't
really hard to catch on...

~~~
eru
But he would have seen it earlier.

------
nebula
A site that is designed to serve a fresh card everyday, keeps serving the same
card.

How is that unimportant? small and trivial technically, yes; but unimportant?

~~~
patio11
Long story short: I thought "a new card every day!" was an important hook when
designing the site. User behavior indicates otherwise -- most don't want
novelty, they either want a) the same card everyone else wants (because it is
seasonal) or b) a particular card which they Googled for. Only about 8% of my
users even see the "featured card of the day" page.

So yeah, on the face of it, this particular feature breaking is unimportant to
most of my users. However, Googlebot is MUCH more impressed with novelty than
92% of my users. Given Googlebot's outsized impact on my business, it is sort
of an important stakeholder :)

