
Facebook makes HipHop for PHP open source - ivankirigin
http://developers.facebook.com/news.php?blog=1&story=358
======
tlrobinson
_"HipHop for PHP isn't technically a compiler itself."_

...

 _"HipHop programmatically transforms your PHP source code into highly
optimized C++"_

Sounds like a compiler to me. Compilers don't have to target machine code.

~~~
DrJokepu
Exactly my thoughts. To further support your statement, theoretically, it
should be possible to create a CPU that interprets C++ source code as machine
code (not sure why would anyone do that though.)

~~~
EricBurnett
By that logic then PHP itself is also assembly because someone _could_ make a
machine that executes it natively. We might as well throw all distinctions
away if we go down this road.

~~~
DrJokepu
I'm not sure if I understand what you're trying to suggest. Indeed, someone
could make a machine that executes PHP natively. It's a Turing-complete
language, just like C++ or x86 assembly for that matter so it should be
possible to build a (weak) universal machine that executes them directly.

~~~
EricBurnett
My point is that what someone _could_ create shouldn't have any bearing on the
discussion. They are using c++ as an intermediate language, fine. Just because
we could make a machine that executes it shouldn't alter that distinction. We
can still call that compiling, but hypothesizing new machines doesn't help.

------
nir
Remember when the meme was "%90 of web apps will not reach a point where
scaling is an issue"? Well, of those who do reach it, %90 will not reach a
point when PHP's execution (assuming you use on of the existing code
accelerator, all which are free IIRC and one will be bundled with PHP6) is the
bottleneck.

It's a cool hack, but really targets Facebook-level needs. For most users, the
downside (eg compatibility issues) will far outweigh the benefits.

~~~
rythie
Actually it's an efficiency gain, rather than scalability. Facebook already
have a system they can add more servers at, but this is designed to let them
use less and save money as a result.

Also at least 90% of web apps are tiny or are startups that never made it,
either way they don't need this for a long time, like you say.

~~~
nir
efficiency -> scalability, in the sense that serving the request faster will
free the server to serve another request.

My point is that for nearly all apps, executing PHP is very small part of the
total request handling time, so even a significant improvement will have
little overall effect. I bet many people excited by HipHop could get a much
bigger performance gain by adding some simple caching)

~~~
rythie
Not really, that type of efficiency is not predictable.

From Wikipedia: "scalability is a desirable property of a system, a network,
or a process, which indicates its ability to either handle growing amounts of
work in a graceful manner or to be readily enlarged. For example, it can refer
to the capability of a system to increase total throughput under an increased
load when resources (typically hardware) are added."

~~~
nir
I realize we're nitpicking, but I don't really get it: Why isn't it
predictable (in the sense that any execution time for an HTTP request ever
is)? You run http_load for x seconds and see how many requests got served,
then make change y and do it again. Obviously there are other variables but
it's possible to get a fair idea.

~~~
rythie
Because as you described you have to try it to find out how much it will save
you, by which point you could probably just roll it out already.

Say you have 10 servers now and you expect business to grow by 10x in next
year, so you put 6 months of dev. time into implementing caching. Then you get
10x the traffic in a month and you get V.C. investment as a result. You then
have a problem, people are getting a bad experience and the development won't
be done for another 5 months. Your CEO buys a rack full of servers, but you
can't do anything with them because you don't have a scalable system.
Alternatively you could find out after 6 months that you can't make that 10x
performance jump after all.

Twitter had this problem a while back, essentially their DB server got
overloaded but it was already a 8 CPU, 64GB machine and despite having lots of
money they couldn't quickly solve the problem, because you couldn't really get
a bigger system.

Facebook knows how to scale, they want to cut costs where they can to become
(more) profitable.

------
KrisJordan
Some details not in the official announcement:

Currently works off of the PHP 5.2 language specs

Compiles to a stand-alone libevent-driven web server or CLI executable

Not based on Zend's runtime/code

Currently no windows support

Intended for 'drop in' PHP replacement - if it runs on PHP and doesn't use a
few things like eval() it will compile with HipHop

~~~
tlack
Did they reimplement the entire standard PHP library bug-for-bug in C++? I'm
curious about that aspect of it..

~~~
JoachimSchipper
Why would they need to do that? The standard library is a mix of PHP, which
they can already handle, and C, which doesn't have to be rewritten.

~~~
tlack
First, they didn't call the PHP standard library: the poster of the root
comment we are replying to has stated they didn't use the Zend runtime or
code.

The PHP standard library has a huge number of functions that are mostly
useless permutations of one another. As opposed to, say, Erlang's meager but
usable lists and string module, PHP has such high-level meta-randomness as
stristr() (case-insensitive search), str_word_count(), and str_shuffle(). So
if you were going to convert that directly to C++, there wouldn't necessarily
be a direct replacement in strings.h. Therefore I'd guess they either have to
reimplement all of those in PHP, and compile that to C++, or write them in
C++. Furthermore, some of these functions have unusual quirks, so some kind of
framework would be required I think.

~~~
JoachimSchipper
I'm sorry, that's indeed what the root comment said. Thanks for the
correction.

Still, I imagine they must use a lot of the same code - half the PHP standard
library is just a thin wrapper of some C library, and reimplementing all of
them sounds stupid.

Then again, they may just not support the majority of such libraries.

------
Quarrelsome
So it converts PHP into C++?!? Woaw.

Didn't people accuse Joel Spolsky of jumping the shark for doing that kind of
thing (Wasabi script => php or vbscript).

.....wait this means that Wasabi can now technically convert into C++! :D

~~~
mseebach
I think where most people chokes on Wasabi is the perception of compiling from
an "insane" language to a sane language. No matter how cool it is to work at
Fog Creek, it's very 1965 to work in a proprietary language, especially for
web development. If Joel wasn't famous, anyone leaving Fog Creek would have
serious problems finding a new job.

Facebook compiles from a "sane" language (PHP) to a less sane, but twice as
fast language (C++). The alternative was to get their devs to work in C++,
which they decided was an unattractive solution -- this seems reasonable.

~~~
acangiano
> _from a "sane" language (PHP) to a less sane, but twice as fast language
> (C++)_

C++ is much faster than PHP, not just twice as fast.

~~~
jwr
Languages aren't "fast" or "slow", it's programs that can be fast or slow.

If a higher-level language lets you implement a smarter algorithm, your
program will run faster.

~~~
acangiano
> _Languages aren't "fast" or "slow", it's programs that can be fast or slow._

We can say that "C++ is faster than PHP", because most, if not all, programs
using the same algorithms will run faster when written in C++ and compiled to
native code.

> _If a higher-level language lets you implement a smarter algorithm, your
> program will run faster._

PHP is generally easier to master than C++, but both are high level enough to
provide adequate tools to write efficient algorithms with comparable amount of
effort, provided one is competent with the language at hand.

------
mrshoe
I see this as being useful if you already have these two things:

1) A large PHP codebase and

2) A CPU-bound app with scaling issues

If you're starting a new app, I don't see why you would choose PHP. And I
think most web apps are database-bound rather than CPU-bound. HipHop seems
incredibly useful for Facebook, but are other people here on HN excited about
using it?

~~~
mikeryan
I think this is where many run into issues with this story. This project was
never really meant to be something to help the PHP community as a whole. It
was meant to be a tool to help Facebook and to that end they've succeeded very
well. 3 engineers + 2 years = 50% CPU improvement without having to rewrite
the whole PHP codebase from scratch? That's a great achievement for Facebook
but YMMV.

~~~
staunch
Seriously. $100k * 3 devs * 2 years = $600k to buy half as many app servers in
the next year(s)? They saved Facebook tens of millions of dollars. I should
imagine their next performance evaluation will be quite smooth.

This is what people mean when they talk about great hackers being 10x as
productive...

~~~
robryan
It would have been a great freedom for the devs, at a company of facebook's
size it wouldn't have been disastrous had they not really achieved a great
enough performance boost to justify it's use.

------
Erwin
This sounds rather similar in principle to Python's Shedskin compiler, which
turns ordinary Python code in C++ code, using C++-based runtime replacing the
ordinary Python runtime. This works extremely very well on a certain class of
problems. <http://code.google.com/p/shedskin/>

------
fizx
Does this make more sense than targeting LLVM? It doesn't seem easier to
maintain, and you won't get the advantages of the optimizations/work already
done.

~~~
alxv
According to the Unladen Swallow's team [1], LLVM's "just-in-time
infrastructure was relatively untested and buggy" when they started working
with it. So, I think not using LLVM was a pragmatic decision. In addition,
debugging C++ code is much easier than dealing with intricacies of dynamically
generated machine code.

[1]: [http://www.python.org/dev/peps/pep-3146/#performance-
retrosp...](http://www.python.org/dev/peps/pep-3146/#performance-
retrospective)

~~~
kingkilr
Yes, but it's not now. Unladen Swallow isn't exactly alone in blazing this
trail, Rubinius is using LLVM as well.

~~~
ighost
Also Lua: <http://code.google.com/p/llvm-lua/>

------
aditya
I wonder if it'll let you convert things out of the box or if it was optimized
for the way php is written at facebook. (eg. it doesn't support eval() but
eval() is bad anyway)

Also, how hard would it be to do the same thing for Ruby and Perl? I'm sure
there have been other attempts at compilers, but as noted elsewhere, having FB
behind it makes HipHop more stable...

Oh, and atleast one thing from this anonymous interview came true:
[http://therumpus.net/2010/01/conversations-about-the-
interne...](http://therumpus.net/2010/01/conversations-about-the-
internet-5-anonymous-facebook-employee/?full=yes)

~~~
daeken
I wrote a PHP->C++ compiler and bits and pieces of the standard library many
moons ago (BinaryPHP: <http://sourceforge.net/projects/binaryphp/> ), and it
really wasn't terribly difficult, even for a complete compiler novice. We
didn't do the fancy optimization that HipHop does, but it worked fairly well
at the time. The actual compiler would probably only take a month or two
(disregarding real optimizations) for a talented developer or two, but the
runtime (assuming it doesn't share anything with PHP proper -- don't know if
this does or not) would take much, much longer.

Doing it for Ruby or Python would be fairly straightforward to get running
initially, but doing Perl<=5 would be impossible (for all cases) due to it
being proven impossible to statically parse.

~~~
viraptor
You can compile any code - full static parsing doesn't have anything to do
with that. If you have a place that is not possible to parse one way, you do a
switch on the condition that causes the parsing choice. Then you compile each
branch assuming that code was parsed in a specific way.

------
maxklein
I really could not justify not doing web stuff in PHP anymore. PHP is THE web
language. No, Ruby is not going to take over, and no, python is not going to
take over.

Those two would have been the main contenders for the space, but they both had
their shot and their 15 minutes of fame, and rather than rapidly growing to
dominate web development, they nichified.

PHP however is being improved, being cleaned up, and most importantly, being
speed optimised.

PHP is what you can trivially easily outsource - try to do that with python or
ruby.

This right here is basically the move that is going to cement PHP as the web
language. The largest sites are using PHP and are going to stay with PHP.
Nothing will take over from PHP. The battle is over.

~~~
ryanwaggoner
I don't agree with you, but I find it sad that your comment has been voted
down below 0. Is this what HN has come to? We just vote stuff we don't agree
with down?

~~~
jonursenbach
That's the point of the karma system.

~~~
jodrellblank
I can't tell if you're trolling, but that's not the point of the karma system;
that being to vote up "good contributions" that you would like to see more of
at HN and vote down "bad contributions" which you would like to see less of at
HN, regardless of whether you agree with them or disagree with them.

------
jrockway
This is truly the PHP solution to the problem. Design a virtual machine that
uses runtime information to generate highly-optimized native code? Nahh....
just regex the source into C++, compile with g++, and enjoy!

(To be fair, some language implementations have achieved good results with
this method; notably Chicken scheme. But I was expecting something better and
potentially more useful to the language implementation community at large.)

~~~
Raphael
C++ is known to be fast, so what advantage could a more complex solution
provide?

~~~
jrockway
Not having to wait fifteen minutes for your app to compile and link, for one
thing. But seriously, the C++ implementation comes at the cost of many of
PHP's features; a "more complex solution" would allow those features to be
kept. And while C++ is fast, a script translated into C++ is not going to be
as fast as if you had written the application directly in C++, unless the
compiler is very, very smart. (I have not seen any implementation that does
this; GHC's native code generator produces faster code than it's "via-C" code
generator, for example.)

Also, static compilation precludes runtime-based optimization techniques,
which has made a lot of code run "faster than C" (including C, ironically; see
LLVM).

So anyway, all this is is a way to compile a limited subset of PHP to C++. In
doing so, you have to write a special dialect of PHP, without PHP's easy
deployability and code/test/debug cycle.

In that case, why are you using PHP to begin with, when a language like GHC or
SBCL would run faster and without any compromises?

~~~
spudlyo
Not even the Linux kernel takes 15 minutes to compile and link anymore. My
desktop machine does it in under 10 minutes, but I realize you weren't being
entirely serious.

<http://xkcd.com/303/>

------
grandalf
It seems odd to me that anyone thinks PHP is "easy".

The syntax is odd, as is the behavior of the built-in functions (many retain
seriously warts from older versions of PHP)

It's "easy" in the sense that you just edit one file and the webpage changes,
but if that's a stumbling block for your developers then you have bigger
problems.

~~~
coliveira
It is really easy if you already have experience with C++. Also, it is not
very different from UNIX shell, and it is clearer than Perl, so I don't get
why people complain so much about it.

~~~
jrockway
Everything is easy if you have experience with C++.

I fail to see how PHP is clearer than Perl. It may have fewer special
characters, but everything is done with oddly-named functions that take
positional arguments in an inconsistent order, return error codes instead of
throwing exceptions, and randomly spew warning messages.

From a language design standpoint... there is no design. Perl may not be
perfect either, but the design has been flexible enough to radically modify as
the community sees fit. Or rather, how _you_ choose to modify it, as most
modifications are modular and only affect a single lexical scope. Oh.

~~~
skorgu
> everything is done with oddly-named functions that take positional arguments
> in an inconsistent order, return error codes instead of throwing exceptions,
> and randomly spew warning messages

These are all anti-features for programmers who already know how to program
but they're quite helpful or moot when you're a complete noob. You're going to
be spending at least a third of your time in the manual so guessable function
names aren't as important as findable ones, you'll be doing lots of "what
the?" style debugging so verbose warnings right on the page save you one fewer
step in the debugging process.

Additionally it's easy to find a PHP hosting provider for peanuts, it will
happily scale to moderate use without overmuch cleverness architecturally and
there are literally thousands of noob-level tutorials that get you from "I
want to build a website" to something that works rapidly. You don't feel the
pain of the language until you're pretty far down the road but as Flickr and
(obviously) Facebook have shown it's not too difficult to limit the pain and
keep growing.

------
icey
Hmm, this is one of those rare occasions where something has been released
where I have a domain name that could be useful - hiphoperator.com

Now I just have to figure out what to do with it :D

~~~
ohashi
at least 1 youtube video

------
spudlyo
I can't wait to try to try this out on a large PHP codebase like Drupal and
then run some benchmarks. It'd be interesting to see how much faster it is
when compared to vanilla PHP vs opcode cached PHP.

------
codexon
I am sad to see Facebook helping to perpetuate PHP.

On the bright side though, even if HipHop yields a 50% speedup, it is unlikely
to be faster than Python, Javascript V8, or LuaJit.

[http://shootout.alioth.debian.org/u32/chartbox.php?s=eNo9ktu...](http://shootout.alioth.debian.org/u32/chartbox.php?s=eNo9ktuRxTAIQ1viDarC%2FXezYm%2Fs%2FJyBQYBwUkPgJz%2FY5BBevRFqyk4JWuYikWFxUvZ7gHTHKS0HToVG56k2A5Pwnri6MkDswQc7IZBDXeVInZopz9NibfJ0qrYlH1x05dnRSkFN9GkdhB2mZHbdUMfKveKidLKbg1xmJ6h5nfZCCQGUU%2FdzxBrTCyZBD7tg6OpiKFDGD1fnqRkX5e7gnt3YCGOMmteZh7tnBKIeYI69BI8UrBTTf38menHntbL3xZ416AE0Rp2E59AY36Yv7j1Fe%2BSB29YaY1duZlA26wrnX9BsbfPN%2BwOA6XrK&m=eNozMFJwS01SMDIwNFAoNTYCABs7A20%3D&w=eNrLKU3MyizxL7PwzylN9M8qKk2q9K9MLCrzL6gsycjPM4bS%2FgUZBf4FqUU5%2FiAVAD2BFWM%3D)

 _edit_

What's with the downvotes? I can't mention my opinions on php?

~~~
ivankirigin
Use your imagination. Perhaps other languages can be compiled down with
HipHop. Perhaps other languages can call PHP libraries compiled with HipHop.
Then you might have an end to end solution to transition from PHP to Python.

This is obviously just a fantasy of mine. Back to coding PHP...

~~~
codexon
> Perhaps other languages can be compiled down with HipHop.

There's already something like HipHop for Python called Cython. You can
compile any pure python code into a static C shared library (with the
exception of generators).

~~~
ivankirigin
I suspect for interoperability, using a single framework is the most likely to
succeed. But this is the whole point of open source, right? Let the community
enhance it in awesome ways.

~~~
mbreese
Are you serious? A magical framework that translates PHP/Python/etc into C++
code? I've got to be misinterpreting what you're saying...

Then again, you might be drinking a bit too much of that Facebook juice... :)

~~~
ivankirigin
Yes, I can easily imagine even people outside facebook adding to the HipHop
effort in adding other languages. Easily imagined and easily executed aren't
the same thing, unfortunately. By the way, I'm out of my element when it comes
to compilers.

~~~
mbreese
I can imagine it really being a very open project with outside contributors,
but I'm not sure that the exact same tools will work with other languages.

For example, I think that dealing with something like Python would require a
completely different set of translations because it is mainly dictionary
based.

That's not to say that doing an X to C++ converter isn't possible or a good
idea, I just think that the HipHop techniques may not apply to other languages
very easily.

------
ricardo_sdl
GitHub were crushed by the release of HipHop. Too bad, I wished to test the
HipHop translator.

~~~
icco
Their file servers crashed before the Hip Hop announcement, and Hip Hop hasn't
even been pushed to GitHub yet, according to this announcement.

<http://twitter.com/github/status/8551643423>

------
mbreese
So, does this mean that PHP has officially forked? I mean who is going to want
to keep writing in the interpreted version as opposed to the hiphop-able
version? How many changes are there? I've only seen mentions of ditching eval,
but there is probably more. Will people be debugging their code in both
vanilla PHP and HPHP?

How well does this support other projects, like wordpress (for example)?

Another issue this raises is who is now in control of PHP? Not in legal terms,
but in terms of mindshare. It's one this for a student to write a PHP
translator, but quite another for Facebook to do it. I'll be interested to see
how Zend responds to this. At least when Google started unladen swallow, they
had Guido working for them.

~~~
qeorge
No, you'll probably see no changes in your day to day life as a result of
this. There's no "interpreted version as opposed to the hiphop-able version",
in terms of the code you write.

As the article mentions, almost no one is really running straight PHP, they're
running it through FastCGI or are at least using an opcode cache like APC. So
you could be swapping out APC for HPHP, but this is at a level of abstraction
below the one you're working in.

Regarding future control of PHP, this does not change anything. Its a compiler
for the language, but does not propose any changes to the language itself.

~~~
mbreese
_you'll probably see no changes in your day to day life as a result of this_

I'm fairly confident about that... I gave up PHP years ago :)

------
agazso
Where is the link to download it or just the documentation? On the
announcement page there is only a Wiki link (besides wikipedia entries and
other php-to-other-lang compilers) to a Github page that redirects to the
github.com.

------
KrisJordan
Some notes from Facebook's UStream event this evening:
[http://www.recessframework.org/page/notes-from-facebooks-
hip...](http://www.recessframework.org/page/notes-from-facebooks-hiphop-for-
php-debut)

High points:

the performance numbers were against PHP _with APC op-code caching enabled_

eval, create_function, and preg_replace with /e (basically eval), and
conditional flow based on the existance of functions are the only features not
supported.

Currently PHP5.2 with a roadmap of 5.2.12 compliant in the next month and 5.3
being the focus after that.

------
euroclydon
What CGI mechanism are they using to host a blended PHP/C++ code base?

~~~
KrisJordan
"It has it's own built-in web server when you compile it, via libevent." from
Ben Ramsey who attended the demo at FB Campus.

------
pmjordan
_With HipHop we've reduced the CPU usage on our Web servers on average by
about fifty percent, depending on the page._

Hmm. I'm guessing they're comparing it to their PHP stack that has already had
the living daylights optimized out of it (with cached bytecode, etc.) and/or a
large proportion of their pages doesn't hit much code (lots of stuff cached?).
I'd have thought there would be more mileage in this.

------
0wned
After this, can we still make fun of C++? I use it daily and love it. Sure, it
takes someone with a brain to use it, but once you learn C++, nothing else
comes close for speed and expressiveness... seems FaceBook just verified that.
So while people who cannot code in C++ will still talk about it as if they
know it, it continues to kick ass in solving real-world computing problems

~~~
tptacek
Facebook appears to be using C++ the way C++ devs use assembly. We can
definitely continue making fun of it.

------
blasdel
How templatey is the generated code? How big are the resulting binaries?

Amazon's original core platform (obidos) was written in C++ and deployed as
one monolithic binary. I've heard that one of the major things pushing their
Java rewrite (dp) was that the binary no longer fit comfortably into memory!

~~~
prewett
Just curious, but how is Java going to solve that problem? Now you've got the
Java opcodes, plus any JIT information/native code, plus the JVM. Seems like
that would be even larger!

------
gojomo
Will Wikipedia, Wikia, and other major MediaWiki sites use this? (Does
MediaWiki code use eval()?)

~~~
Perceval
I would bet that Wikipedia is more database bound than CPU bound. HipHop would
probably help, but not tremendously.

~~~
sailormoon
You reckon? I would have thought Wikipedia a textbook case for a great big
cache. I would have guessed that the great majority of their views were read-
only by unregistered users, with the pages in question following a power law
distribution. Hm, would be interested to find out more about that.

update: I went and looked it up. They do indeed have a great big cache, many
of them in fact. In fact their Amsterdam presence runs _only_ cache! From what
I can tell, it seems their bottleneck is actually Apache, and presumably a lot
of that is PHP.

Interesting presentation here:
[http://wikimania2009.wikimedia.org/w/index.php?title=Media:R...](http://wikimania2009.wikimedia.org/w/index.php?title=Media:Rob_Halsell_-
_Wikimania_2009_-_Wikimedia_Operations_%26_Technical_Overview.pdf)

Also check this out:
[http://ganglia.wikimedia.org/pmtpa/?gw=fwd&gs=Wikimedia%...](http://ganglia.wikimedia.org/pmtpa/?gw=fwd&gs=Wikimedia%40http%3A%2F%2Fganglia.wikimedia.org%2F)

Go down to the MySQL section. Plenty of headroom, but the Apaches are working
hard.

------
ojbyrne
So... stupid question - Where's the code? It's the next day and no sign of it
on github.

------
carson
It would seem they forgot to make the project public on github.

~~~
andymism
Third paragraph from the bottom of the post says they'll be releasing the
source in Github this evening (probably pacific time).

~~~
carson
Ahhh, they should have held off on that wiki link.

------
regularfry
If this makes a big difference to memory footprint, won't it be rather handy
to embedded devs?

------
evlapix
If this is adopted widely hosting costs will drop off considerably, no?

------
peter_o
Will this be faster than less say... python?

------
erickerr
They probably just use a python script and type

from php import cppCompiler

not sure why that took 2 years

------
h0h0h0
Damnit - a cool project name wasted on a php to c++ compiler?? DJ Kool Herc is
disgusted with Facebook.

------
runT1ME
Facebook's lack of engineering prowess continually amazes me.

There were quite a few other options aside from writing a translator.

~~~
sailormoon
Are you living in an alternate universe? Facebook are scarily competent. They
have executed brilliantly and their engineering is top notch.

~~~
runT1ME
Really? You mean like their ultra-reliable chat server they wrote?

Or maybe the way their replication works, so when you post things get out of
order depending on what server you are looking at?

~~~
sailormoon
With criticisms like that, I think your expectations are unrealistic.
Considering the size of their userbase, their chat server is quite an
accomplishment. And minor replication artefacts are hardly a big issue.

In short, if FB had incompetent or even "just" competent engineering, they
would not have close to 400 million users. And yet they do, and I have never
noticed the site performing slowly. They're not Google, fine, but they have an
excellent team and if you're not impressed by their performance I wonder what
you would be impressed by.

~~~
runT1ME
Having a large user base is not an indication of technical authority.

Myspace had (and still does to a lesser extent) extremely large user base, and
we know they had some stability issues (Honestly though, I've listened to
their tech presentations and am more impressed than the things I've seen
Facebook do).

No, I don't imagine Gacebook is programmed by a bunch of incompetent monkeys,
but compared to Google, Amazon, Yahoo, or many other companies that have had
to deal with large scaling issues, their approach seems amateur.

------
icefox
"300,000 lines of code and more than 5,000 unit tests."

Both values sound good in theory, but are both meaningless. How is the code
coverage? How many man months? How many developers?

~~~
henning
I find it ironic that you feel LOC is meaningless and then asked for man-
months, an even more useless unit of measurement :D

