
As Digg Struggles, VP Of Engineering Is Shown The Door - icey
http://techcrunch.com/2010/09/07/digg-struggles-vp-engineering-door/
======
amix
Like Joel Spolsky said it: "They followed the single worst strategic mistake
that any software company can make: They decided to rewrite everything from
scratch" - <http://www.joelonsoftware.com/articles/fog0000000069.html>

I think rewriting from scratch is the core of their problem and not really
Cassandra. Gradually going over to Cassandra would have been a much better
idea.

~~~
squidsoup
I've heard this time and time again, but what if your application is a genuine
ball of mud? Would you really not advocate a rewrite for an unmaintainable
spaghetti classic asp app still in production today?

~~~
_delirium
Microsoft could be one example of a successful one. Their flagship Windows
codebase that ran the 3.1->95->98->ME line was basically ditched with the
rewrite-from-scratch NT (famously done by an ex-VMS team), which later had
some APIs ported to it to make Windows 2000 and especially XP be close to
drop-in replacements for the old line, while not really sharing much code. I
think in retrospect that was probably a good idea: the NT rewrite put the
codebase on much better footing than the aging, incrementally updated classic
Windows codebase had been.

Solaris is another example of a rewrite that seems to have worked, though the
rewrite did derive from a different set of existing code, not a total from-
scratch job. But the classic SunOS 1.x.-4.x codebase was ditched, and SunOS
5.x / "Solaris 2" replaced it.

~~~
alabut
You covered Windows and Linux - don't forget Apple's switch to OS X. They
wouldn't done a wholesale switch at some point even if they hadn't gone with
something unix-based because the other alternative was Copeland, the internal
project to do a complete rewrite.

~~~
alabut
EDIT: they _would have_ made the switch. Stupid iPhone.

------
cookiecaper
Is there any reason that Cassandra is the focus of this article? It is really
silly and irresponsible to peg a nascent project like that without any
reasoning or sources. I'm sure something changed besides just a Cassandra
rollout, and wasn't Digg using it on v3 too?

I think Cassandra is pretty well tested. There have been lots of super-large-
scale deployments. It just seems lame to blame it on that, but I guess maybe
their anonymous sources inside Digg revealed it? But then we'd hope they'd
know if the problem was with the datastore or the implementation.

~~~
sliverstorm
Placing all the blame on Cassandra helps Digg justify letting go the VP, since
he was allegedly the guy who pushed it.

~~~
TheCondor
So what's the digg story here?

I get that some VP suggested a new buzzwordy technology, they gave him enough
rope to hang himself and he did and left the company with a broken pile of
crap. It could happen, if you have a healthy company you give trust to people.
That it got this far doesn't speak well of the rest of management. It doesn't
speak really well of the rest of the team either. Shouldn't there have been
some circuit breakers or something?

Digg isn't a poor, bring-your-own-laptop startup. They've got resources,
they've had substantial investment. They can afford to build and test software
and I know of no real marketing reason they had to push something untested
out. Rose could go out and say it wasn't done, it's going to take more time.

How does Rose keep his job? Wasn't he this VP's boss?

And it's single technology? That couldn't have been vetted and tested
independently of all of digg? Really? And MongoDB, or hadoop, or one of the
dozens of other nosqls wouldn't work, either? You do what you have to do and
there is never a 'truth' with VPs and CxOs are canned, but it all doesn't
float with me, just looks like another over valued and under talented company,
got lucky and the blind squirrel found the nut, there isn't gonna be a second
act.

Maybe it just seems low brow to me, name and finger the guy, blame the
opensource tool you use, never explain or elaborate why you launched anyways
when you were fixing bugs in the tool at the 11th hour.

~~~
ojbyrne
I'm sure the real story is different. My rough guess at it is

* the main marketing tool that digg has left is "Kevin Rose as genius"

* Kevin and the people he drinks with pushed for the adoption of new cool technology.

* the entire thing was a giant clusterfuck because of Kevin and the people he drinks with.

* but once that became obvious there was a need for a scapegoat so that digg could keep its primary marketing tool.

~~~
dillydally
That reads like sour grapes.

~~~
binaryfinery
Reads like "industry experience" to me.

~~~
nailer
Perhaps it's both?

------
mikeryan
I actually wondered if they were running into issues with Cassandra. I'm not a
NoSQL hater - but its still pretty bleeding edge and it always seemed like
making it your core db was super risky.

arg - really want more insight, maybe Quinn will elaborate now that he's gone.

------
nl
Digg's put out a lot of information about their Cassandra implementation, eg
<https://nosqleast.com/2009/slides/sarkissian-cassandra.pdf> (from November
2009).

Back then they seemed to have a rather sensible migration strategy (ie,
basically running the new Cassandra back-end in parallel with the MySQL
backend).

It seems to me that it was the v4 upgrade that broke, not Cassandra alone.
It's possible their frustrations with Cassandra were more long term, and the
fact the v4 upgrade didn't go well was the last straw.

For example, Digg has done a lot of work on Cassandra internals and tools. If
you are using a new, open source product you kind of expect that, but it's
possible the expense of that didn't seem like good value once v4 started to
get into trouble,

------
btipling
For some reason this reminds me of the "no one has ever been fired for
choosing IBM." slogan. I guess maybe there is some truth to it. Seems like new
tech needs to start slow on big places, only startups really have the freedom
to risk it all.

~~~
aaronblohowiak
your post implies digg isn't a startup.

~~~
soofaloofa
Digg isn't a startup. It started in 2004 and has been mature for about 5
years. They are a business now. And a big one at that.

~~~
Silhouette
That depends on how you define your terms. I'm not sure a company that has
been around 6 years and has ~100 staff can be called a startup any more. On
the other hand, I'm not sure a company that still survives more on the wishful
thinking of investors than on the money it brings in after 6 years can be
called a business, either, and certainly I wouldn't call it either big or
mature. I'm honestly not sure what I _would_ call that sort of organisation,
though if I were an investor, I suspect the word "liability" would feature
somewhere.

~~~
rbanffy
> I'm not sure a company that still survives more on the wishful thinking of
> investors

Unless you are owned by a larger company that runs you at a loss as part of
their strategy. Or stragedy.

------
agentultra
This man's reputation is on the line. I hope they release more details on what
exactly it is that is causing the problem. As it stands he appears to be an
unfortunate scapegoat.

~~~
heffay
Hopefully he creates a blog post explaining the disaster. He seems to
regularly update that site

------
madridorama
Digg turned into nothing but a paid distribution and SEO engine anyway

~~~
blasdel
Digg was something else before?

~~~
josefresco
Yeah originally it was Slashdot but better, more current and user powered
(which was a new concept in the mid 2000's). What it turned into was a traffic
generator for a small group of power users who fed the community the lowest
common denominator of content.

~~~
jshen
User powered will always turn into lowest common denominator if the user base
grows beyond a niche

------
narrator
Reddit runs on Cassandra + PostgreSQL. They use Cassandra as a key/value store
and not as the primary database though.

~~~
cookiecaper
They are planning to transition to Cassandra as the primary database.
PostgreSQL is not used in a relational way on reddit -- it is used as a
makeshift k-v store. You can take a look at this yourself as reddit is fully
open-source: <http://github.com/reddit> .

~~~
zzzeek
> They are planning to transition to Cassandra as the primary database.

do you have a source for that ?

> it is used as a makeshift k-v store

That is very true, they should switch. It just seemed like they had no plan to
do so.

~~~
bbatsell
I could swear an admin said it with some pretty fair definitiveness; I thought
I remembered that it was ketralnis, but this was the best I could find from
him, and it's not nearly as definite as my memory says.

[http://www.reddit.com/r/programming/comments/bcqhi/reddits_n...](http://www.reddit.com/r/programming/comments/bcqhi/reddits_now_running_on_cassandra/c0m3rs9)

------
megaman821
This article is a fine example of the poor logic that pisses me off nearly
every day. Just because Digg v4, which heavily uses Cassandra, can't be used
as a Cassandra success story does not automatically mean you can assume the
opposite, that it is a Cassandra failure story. There is no indication why
Digg has been down so often and there is really no conclusions to draw as of
yet on the technology they use.

~~~
fleitz
The article basically comes right out and says it. You just need to read
between the lines a little bit.

From the article: Quinn was the main champion of moving over to Cassandra, say
our sources. Now the site is taking a huge hit, at least in the short term,
because of that decision and/or how it was implemented, and Quinn is paying
for it with his job.

It's always a toss up between whether it was implemented correctly or not. The
correct course of action of course would have been to slowly move the site
over to the new technology piece by piece rather than a wholesale switchover.
The risk is in the migration strategy not the technology picked. They could
have been equally stupid switching over to a new architecture with mysql.

~~~
jshen
I'm not sure I'd use tech crunch as a reliable source.

------
mcantelon
Reddit went through similar issues a few months back (downtime, slowness,
etc.), but they overcame these issues without turfing people. My guess is Digg
pushed the engineering VP out to make the investors happy rather than to
actually move forward.

~~~
pavs
How does it make investors happy? Why would investors care what kind of
database you use?

~~~
fookyong
it's an attempt to control the PR message.

by doing this, they are laying the blame on one person so the media can stop
hating Digg and start hating the ex-VP of Engineering who "killed Digg".

of course, whether he was actually responsible in some way is something we may
never know. for all we know he may have been completely against releasing v4
but was vetoed by Rose et al. or on the other hand he may have overpromised
and under-delivered, putting the company in jeopardy, in which case he
deserves to be let go.

it's all speculation until we hear an official comment from either side.

------
pclark
Unless Digg cannot roll out new features due to Cassandra (eg: a) Cassandra
woes taking all dev team time, or b) tech limitations prevent features) it
seems highly unlikely that cassandra is the reason why Digg 4 is grating with
users initially.

It's a change of features/product rather than technology that is the problem.

~~~
enjo
I think the folks at Digg where prepared for the heartburn associated with the
content changes. They weren't ready for some very serious issues with the
implementation. Digg has definitely seen a lot of downtime over the last week
or two.

------
heffay
I'm really interested to see what comes of this and what went wrong. It sounds
like (from reading his blog) they were making a lot of customizations to
Cassandra?

~~~
pquerna
They have made many feature additions to Cassandra.

They haven't said much about the details of why they are having trouble.

It could be a core Cassandra problem, something they added, or completely
unrelated to Cassandra; But the internet doesn't care. It's drama at its
finest.

~~~
ergo98
"It could be a core Cassandra problem, something they added, or completely
unrelated to Cassandra; But the internet doesn't care. It's drama at its
finest"

Digg made a _big deal_ about their move to Cassandra (just as Digg's move to
Cassandra was used to legitimize Cassandra, and by correlation NoSQL, among a
wide range of zealots), going back over a year.

The thing about talking big like that is that it often comes back to bite you
in the ass if things don't go well.

If Digg quietly released a new version that worked more reliably and provided
a better experience, they would have been in a perfect position to pontificate
on technology.

------
mminolt
Ok Digg is having a hard time. I feel like I witness a strange blood thirsty
tone in some of the comments though. I wish Digg the best and I hope they can
prove all the skeptics wrong. It's going to be tough though!

------
houseabsolute
I wonder what issues they are running into that a dark launch would not have
found. As I have discovered painfully in my own projects, making big changes
without a rollback plan is usually a bad idea, and it sounds like this is no
exception.

~~~
gphil
Yeah, it sounds like the launch process is more at fault than the technology
for this if there's really no way to rollback to stability.

------
zephjc
Digg v4 has added back the features I liked after taking them away - mostly
the Upcoming section, which tends to have interesting things that never make
it to the front page, and the setting that allows you to default to Top Mews,
instead of things I have already read ('My News').

For most users, change in features is what drove them away, not necessarily
the spotty QOS (though that was pretty bad for a while).

------
parfe
I have to say I like the new Digg. Sad this guy is getting his career stomped
on over it. Lot of people complain that the new site lets big sites submit
their own content automatically, but how was it any different with a middle-
man user submitting it himself?

~~~
gamble
The real problem is that v4 removed the illusion that average users had an
influence on the rankings, without altering the fact that they don't.

~~~
hop
Yeah, I may be blind, but how do you even get to the upcoming stories?

~~~
gamble
You can't right now. It's on the list of things they haven't finished
reimplementing from v3.

------
Dervish
Digg 4.x is like Star Wars Galaxies NGE.

------
c00p3r
Let me guess - the problem is about the difference between theory and
practice.

In theory, Java is great and Cassandra is great. In practice - Java under a
heavy load is a disaster, because it was never designed for it, and Cassandra
is a just a hype and propaganda.

Face the reality - it doesn't work in production as it supposed to - as a
primary storage engine.

People at the Digg aren't amateur idiots, so I think they do everything as it
described in docs, but the damn thing just doesn't work.

~~~
nl
> In practice - Java under a heavy load is a disaster

Google's heavy use of serverside Java would indicate otherwise.

> because it was never designed for it

Yes it was. Java has a lot of problems, but one thing that isn't a problem is
heavy load.

> Cassandra is a just a hype and propaganda

Facebook seems to be using it ok.

~~~
c00p3r
Google probably use server side Java without a really heavy load, and just add
more servers to keep the load low.

System which was designed for being isolated from an OS (leave alone hardware)
will have a bottleneck exactly in this level.

Facebook doesn't use it as a primary storage with a high load.

~~~
cameronh90
What would you prefer to write web applications in? C++?

Most languages used in web programming are divorced from the hardware, and
either use a virtual machine (e.g. Java, .NET) or an interpreter (PHP, Python,
Perl, Ruby).

Virtually no-one uses a low level language for web programming, and the
benchmarks say that the virtual machines are faster than interpreters. Java is
very fast and efficient once it's running, but initial start-up time is often
slower than interpreted languages due to the JIT compilation, but servers
rarely "start up". If you're really hitting a barrier with Java's performance,
just as with other non-native languages, you can write the performance
critical section in C.

[http://shootout.alioth.debian.org/u32q/which-programming-
lan...](http://shootout.alioth.debian.org/u32q/which-programming-languages-
are-fastest.php)

------
rbranson
Queue NoACID haters declaring this the final nail in the coffin.

~~~
dangrossman
This isn't Digg.

