
Facebook’s code quality problem (2015) - setra
https://www.darkcoding.net/software/facebooks-code-quality-problem/
======
combatentropy
I look at programming as all design, whether deliberate or not. Martin Fowler
drove the point home for me
([https://www.martinfowler.com/articles/newMethodology.html](https://www.martinfowler.com/articles/newMethodology.html)).

He said that some people drew inspiration from construction, where there are
designers and builders. One or two highly paid architects draw up the plans.
Then you can hire a bunch of cheap labor to build it, say a bridge. This
belief leads to a dichotomy in software companies, where one person is the
"architect" and others are just "regular" programmers --- often outsourced to
the lowest bidder.

From Jack Reeves he cites the epiphany "that in fact the source code is a
design document and that the construction phase is actually the use of the
compiler and linker." There is little repetitive, mindless work in
programming, because as most of you know, "anything that you can treat as
construction can and should be automated." Therefore, "In software all the
effort is design . . ."

~~~
krick
I feel your sentiment, and usually oppose to the construction-driven approach
you describe, but it is really easy to play devil's advocate here. I mean,
let's just be realistic. There's nothing sacred in programming.

The fault in your argument is that there's no design/construction dichotomy,
but rather a gradient: everything can be viewed upon as a design from close
enough distance, and everything can be a construction looking from the high
above. For business people all the programming is just very, very low-level
construction, and there's no design to it. Damn, even hiring programmers and
creating companies is just a construction stage to them.

And from the practical standpoint, the truth is that perceiving a lot of
programming "just as a construction" just works. For the better or worse. It
is cheaper and usually reliable enough to design the system in a way that a
lot of the components would be plugins not depending one on the other that can
be outsourced to anyone. Even if some fail, the system continues to work while
you hire new people to rewrite some components. Sure, maybe it would be more
efficient and "cool" to hire a really smart guy (or do it yourself, if you
believe you can) to write it as really optimized monolith system, but if this
isn't required, it is usually less risky to build something not so
sophisticated, and improve it if need be. It is usually ok, that some parts of
the system may fail. Fuck it, let's not lie to ourselves: there always _are_
some parts of the system that fail, whether you think it's "acceptable" or
not, let's just take it into account.

------
paganel
I read Richard Gabriel's "Worse Is Better" essay in my first month hired as a
professional programmer, that was almost 12 years ago. Since that time the
lessons of that essay have only increased in significance for me, with this FB
story as additional proof.

Their back-end code might be bollocks, and I can certainly believe that
judging by how sluggish their FB app feels on my phone, but the fact is that
they've conquered the Internet (together with Google and a couple of other
companies). It's a fact that I personally hate, but they're still winners in
the end.

~~~
aaron-lebo
But it's not better, it is worse as clearly put forward by this article. ;)

We shouldn't attribute Facebook's success to poor quality code, unless we
really believe that 429 people working that that slow sluggish app is
productive. Do you?

Maybe Facebook would be in a better situation than they are if they had a firm
foundation? What happens when a competitor comes along and knocks on their
door? That mess of code becomes an anchor.

They succeeded not because "worse is better" but because ...they succeeded,
maybe because they had the greatest social network in the world to expand
within, or because of some other reason we don't know. But they didn't succeed
because of worse is better, they succeeded in spite of it.

~~~
jaredklewis
No one is actually arguing that worse is better. It's just a catchy title. A
more accurate title would be "worse is faster than better, and being fast but
mediocre is better than being slow and amazing."

Worse is better makes the observation that doing right thing slows you down.
Often, an ok solution is much quicker to get finished than a great one. And it
concludes that the projects that prioritize getting stuff done over code
quality will suceeed.

Would facebook be as successful as it is today if it had taken longer to ship
features, but maintained higher code quality? Who knows.

But the intuition in worse is better, which matches with my own experiences,
is that making that trade off would have cost Facebook, as other faster moving
projects closed the gap.

~~~
polack
I agree with you that it's sometimes good to focus on getting things out the
door fast instead of the code quality behind it. But surly there must be a
point when the pendulum swings back and you should switch focus to quality. I
would argue that Facebook have definitely reached the point where they own so
much of the market and with their infinite resources they should be able to
ship things with good code quality. Sooner or later the costs of supporting
all that crappy software will make a dent even in Facebooks chest of gold.

~~~
collyw
They have written their own PHP interpreter haven't they? Thats a pretty good
indication that they left some things too late. Saying that, if they have the
resources where that sort of decision makes sense, then its probably not going
to be too much of a dent in their pot of gold.

------
charleslmunger
"Exhibit C: Our site works when the engineers go on holiday"

Is not all so different from "Fewer patients die when heart surgeons are on
vacation". Of course your site is going to be more reliable when nobody is
changing it! You should be worried if reliable isn't the steady state, and it
requires constant changes to stay up!

~~~
afc
I also took issue with this part of the article; it seems obvious to me that
rolling out new features would directly cause reliability issues (and the only
alternative, of not rolling out new features, is, obviously, a cure worse than
the disease).

~~~
adrianratnapala
I am not sure that most Facebook users would actually agree about this.

But then their view is slanted towards UI changes. Performance improvements
and other subtleties are features too.

------
thedevil
It's hard to argue with the business sense of pushing out features quickly at
the cost of code quality.

But I've learned that that there are two kinds of development teams:

(A) the teams that are "moving fast and breaking things" while "creating
business value"

and

(B) the teams that are "only maintaining legacy code", "not getting anything
done", "costing a lot of money" and "complaining too much" about code quality
that team A wrote.

As an engineer, I've learned that it's less work and more rewarding to be on
the (A) teams than on the (B) teams.

~~~
sAbakumoff
I like type (C) whose motto is "move fast and fix things"
[https://githubengineering.com/move-fast/](https://githubengineering.com/move-
fast/)

~~~
ricardobeat
And yet [https://github.com/blog/2106-january-28th-incident-
report](https://github.com/blog/2106-january-28th-incident-report)

~~~
Nition
I like how even the URL has a "bug" (year 2106?).

~~~
amnah
uh no. that's the id of the blog post

[https://github.com/blog/2105](https://github.com/blog/2105)
[https://github.com/blog/2106](https://github.com/blog/2106)
[https://github.com/blog/2107](https://github.com/blog/2107)

~~~
Nition
Aw, it looked so much like a minor mistake with the Y-M-D date format
considering it happened on 2016-january-28th!

------
zo7
Some of FB's ~18,000 Obj-C header files from a post linked in the article: [1]

* FBFeedAwesomizerProfileListCardViewControllerListenerAnnouncer.h

* FBBoostedComponentCreateInputDataCreativeObjectStorySpecLinkDataCallToActionValue.h

* FBEventUpdateNotificationSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h

* FBGroupUpdateRequestToJoinSubscriptionLevelMutationOptimisticPayloadFactoryProtocol-Protocol.h

* FBMemReactionAcornSportsContentSettingsSetShouldNotPushNotificationsResponsePayloadBuilder.h

* FBProfileSetEventsCalendarSubscriptionStatusInputDataContextEventActionHistory.h

* FBReactionUnitUserSettingsDisableUnitTypeMutationOptimisticPayloadFactoryProtocol-Protocol.h

oh my god

[1]: [http://quellish.tumblr.com/post/126712999812/how-on-earth-
th...](http://quellish.tumblr.com/post/126712999812/how-on-earth-the-facebook-
ios-application-is-so)

~~~
aljones
Those read like classes generated by tools. Or maybe I really don't want to
think humans are capable of such names.

~~~
oriolid
These are completely credible class names, just ask any Enterprise Java
developer. I particularly enjoy FactoryProtocol-Protocol. Who would be
satisfied with just a FactoryProtocol?

~~~
ryandrake
Yea, I was going to say, Java's almost got them beat with
SimpleBeanFactoryAwareAspectInstanceFactory[1]

1: [http://docs.spring.io/spring/docs/2.5.x/javadoc-
api/org/spri...](http://docs.spring.io/spring/docs/2.5.x/javadoc-
api/org/springframework/aop/config/SimpleBeanFactoryAwareAspectInstanceFactory.html)

~~~
ReidZB
Invoking the Spring framework here is just cheating -- they have dozens of
classes like that! :-)

Just the other day I was dealing with Spring's declarative transactional
support. Here's a fun read: [http://docs.spring.io/spring-
framework/docs/4.2.x/spring-fra...](http://docs.spring.io/spring-
framework/docs/4.2.x/spring-framework-reference/html/transaction.html)

------
fishnchips
I spent a few months at Facebook and left because of that. Not necessarily
because of poor code quality but because of all the issues it was causing, and
a culture of turning a blind eye to (or worse still, being proud of!) those
issues. It's obviously true that quality of their code does not affect their
market position, but it sure affects who they hire and who they retain.

~~~
joeletizia
That may be true. But in the end, does it matter?

Until we see someone taking market share away, I'd argue it doesn't. Facebook
still retains top tier people regardless of any perceived quality problems.

Sorry you left. I just started here, and I do agree that what I feel about
code quality isn't generally shared here. But as I get older, I start
realizing that in the end, as long as things work relatively well at our scale
and in our problem domain (we're not medical device software, we can fail)
alls well.

~~~
Sevii
It matters because profits are lower. 400 devs on an iOS app is not free.

~~~
joeletizia
But FB is a public company. If shareholders thought that was the case, they
could vote with their dollars. Looking at the stock price, they seem to have
no problem with the way FB is hiring engineers. The market is okay with it.

There's no reasonable way to say FB should have less engineers without being a
high level director of the company.

~~~
xaa
> There's no reasonable way to say FB should have less engineers without being
> a high level director of the company

I'm going to assume you don't literally mean only company directors can levy
logical criticisms of a company.

Consider the fact that the FB app itself was built with a very small team to
begin with and only once it became 80-90% feature-complete did the exponential
growth in engineers suddenly become "needed".

I can understand why Microsoft needs tons of engineers. OSes are complicated
and have lots of moving parts (drivers, file systems, kernel, etc) that are
all absolutely required and must be constantly updated, even without bringing
their other products into it. But Facebook? At its core, it's a single webapp
that serves simple relatively static pages. Yes, at web scale, so it's not
entirely trivial.

My explanation is that Google and Facebook face a similar dilemma: most of
their revenue comes from a single, relatively simple core product. To really
optimize that core product, they might need a couple of dozen highly skilled
engineers, along with a lot of equipment to scale it.

But that's not why they hire thousands. It's because they're both constantly
trying to branch out for growth. In terms of revenue it is fair to say that
strategy has been a failure for Google and except for a few of the "bells and
whistles" FB has added like IM, probably for them as well. This problem is
caused by the way that tech companies are primarily valued by perceived growth
potential rather than actual earnings. They can get away with hiring lots of
useless people because capital costs in this industry are fairly low.

------
userbinator
It is ironic that a codebase with so many classes is likely to occur precisely
because of dogmatic adherence to those "best practices" which were originally
intended to improve code quality --- aggressive refactoring/modularisation
being a likely culprit. What I think they really need is not more design, not
more architecture, not more of anything but KISS and YAGNI.

~~~
afarrell
The thing that having an overall architecture buys you is conceptual clarity.
That lets someone much more easily understand what is going on. You need to
have a minimal set of ideas that are well thought out and explained.

Just saying "YAGNI" and "KISS" doesn't actually make the resulting code
simpler. Why? You do have _some_ things to build and refusing to lay out a
plan for what that is going to be doesn't stop that. It just means that the
complexity doesn't have a clear narrative

------
dawidloubser
I would like to argue that, in most of these cases, it's not "code quality"
that is at fault, but "design quality" (before coding) - which is often absent
entirely.

The problem with design, in software, is not that most people forget to do it.
It's that they never learn to do it. It always comes back to bite you.

I don't want to start a discussion on design, and how most people mess it up
because of lack of skill or experience therein. But hacker culture seems to be
allergic to design, and hacker culture seems to be what everybody strives for
these days.

~~~
lefty2
I think you hit the nail on the head. My motto for developing code is "make it
simple as possible, but no simpler". Whoever, created a design that used
18,000 classes did not follow that rule.

------
whack
I do think that FB, and most large orgs, have code quality problems. But the
article does a pretty bad job at making its case.

> _" That’s 429 people working, in some way, on the Facebook iOS app. Rather
> than take the obvious lesson that there are too many people working on this
> application, the presentation goes on to blame everything from git to Xcode
> for those 18,000 classes."_

How does the author know that 429 is too many? How does the author know that
FB's goals and functionality can be best achieved with fewer people/classes.
This just reads like a classic "Why is Google so big, I can do that in a
weekend" comment ([https://danluu.com/sounds-easy/](https://danluu.com/sounds-
easy/))

> _" Our site works when the engineers go on holiday"_

This is pretty much universally true for any dynamic site where engineers are
constantly adding new features. Change always comes with risks. Changing code
is always more likely to break something in the very-short-term, compared to
not changing code. I have a hobby site (shameless plug,
[http://thecaucus.net](http://thecaucus.net)) which has been running in auto-
pilot for the past year, and almost never breaks, because there are virtually
no moving pieces. The fact that the FB site breaks more often when engineers
are making changes to it, is just repeating a universal law of software
development.

I do think that the organizational structures at most large companies are
bloated, inefficient, non-transparent, and produce sub-par code. I had high
hopes when I read the article's headline, but the arguments presented simply
aren't very persuasive.

------
rumcajz
I believe "our site works when the engineers go on holiday" thing is not fair.
Of course that application is less stable when it's actively modified. There's
no way to make it more reliable on weekdays than on weekends except for
stopping the development altogether or maybe deploying on weekends.

~~~
dsr_
No way at all? FB has a lot of servers and a lot of users. They have
opportunities for quality practices that few other organizations get.

Here's a simple example:

Split the user population into 365* groups. Assign them to days of the year.
Test new code only the group whose day has come up. Follow them until they
stop having problems. Now you can deploy that to a month's worth of groups.
All good? Deploy to everyone.

Yes, that means that you can't have more than 365 changes in simultaneous
development. Tough.

*Yes, yes, leap years. Take a day off from deploying.

~~~
byroot
They graph the number of incidents. Even if a bad release impact only 0.3% of
the user base, it's still an incident.

They have to investigate it, revert or fix the bad code and start the
deployment process again.

~~~
rumcajz
Ack. Also, the system introduces a new problem: If you are deploying on
weekend, most devs are not around to help solving the problem. Thus, the
outages would be longer.

------
ptero
Code quality is important, but for an organization the size of the FB good
systems engineering is much more important than good software engineering. If
the overall system is organized sanely, failures of subcomponents (which
happen due to software bugs, hardware, humans, etc.) can be isolated,
firewalled and fixed in reasonable time without bringing the system down.

------
BinaryIdiot
As pointed out this is from 2015 but I'd love to hear some updated. Has any
Facebook employee created newer presentations that discussed, say, their
18,000 iOS classes and the fact that in a single week 429 people contributed
to the same iOS app?

I'd also love to hear an update to the "Our site works when the engineers go
on holiday" claim.

~~~
arjie
I'd bet that the 18000 classes is just Thrift-generated stuff or whatever.

~~~
panic
You don't have to bet, you can see them here:
[http://quellish.tumblr.com/post/126712999812/how-on-earth-
th...](http://quellish.tumblr.com/post/126712999812/how-on-earth-the-facebook-
ios-application-is-so)

~~~
rimliu
Even with a list my imagination is still not powerful enough to understand how
an app like FB needs 18000 classes :(

~~~
mmjaa
It _needs_ them .. to keep all those programmers busy.

------
planetjones
Well when there is a codebase of the scale of Facebook's, with so many
activate contributors of course there will be a code quality problem. I see so
much stuff at Google breaking or becoming inconsistent too (away from the core
search features).

Even if they didn't move fast and break things, they would still have code
quality issues. It's just the nature of a huge codebase like they have. And
let's face facts - the development process needs to be suitable to the type of
problem being solved. This is not a life and death situation and users would
value new features, more than extreme reliability and consistency.

------
mybrid
It is worth noting that when something goes viral we still don't know why.
Facebook represents the ultimate in viral success. Given we do not understand
why things go viral then it stands to reason that the underlying software
methodology is not causation. This means that the fickle finger of fate could
easily be visited upon Facebook the same as with MySpace and Facebook will be
in no control. This is independent of the any software methodology they use.

Claims made without evidence can be dismissed without evidence.

------
aamederen
I believe, from time to time, you just need to stop developing new stuff for a
period of time and dedicate the team to improve the overall quality, pay
technical depts, hunt bugs, remove legacy code and refactor still existing
ones.

~~~
onion2k
And fortunately your competitors are always happy to pause and wait while you
do that.

------
bobbyi_settv
> The second exhibit is from Facebook Research... wouldn’t you just write your
> disk files to a ramdisk? Surely they noticed this too

I'm not sure the author understand what a Research team is and what it does.
Trying out a new solution to an existing problem is sort of their job. I'm
also not sure how a research team publishing a paper discussing an alternate
solution indicates anything about the company's "code quality".

------
amelius
It also seems that improvements in code quality caused by React happened not
because they were driven by management, but because a few "heroic" engineers
thought it was an interesting side-project. At least, that's my impression
from the publicly available talks they gave.

------
watwut
Reality check is that most companies, including most startups, attempting to
create software of such size end up tangled in their own mess way sooner and
fail. Keeping it maintainable at such scale is not as easy as keeping it
maintainable when there are four of you.

------
segmondy
So?

I've become very cynical since the past year or so because there is a lot of
noise/articles on the net, all knowing what's best for you or wrong with what
you are doing. Yet without context or true authority to talk about it. My
question these days when I read such article, "What has the author done that
is equivalent?" There is rarely anything to be found.

Facebook might have a "code quality problem." But until you have worked at an
organization that is big like Facebook please hold forth your tongues or
fingers in this case. Startup principles don't apply nor academia. Facebook's
"code problem" is what really does happen in the real world with such
humongous business enterprises. Lot's of moving pieces in code, in people, in
ideas and the amalgamation of these results in what most programmers will see
as "code quality problem" yet the market sees as billions of dollars.

~~~
aaron-lebo
Isn't that an appeal to authority? Because someone hasn't done X they don't
have the right to criticize X?

 _Facebook 's "code problem" is what really does happen in the real world with
such humongous business enterprises._

But does it have to be this way?

~~~
evgen
Until someone provided s counter-example the assumption is that yes, this is
just what happens. We have entire industries of competing methodologies, some
almost cult-like in their slavish devotion to "the one true process", and none
have birthed any great successes. Why is that?

~~~
chii
Let me provide a counter example. facebook, revenue in 2016 is $27 Billion,
maintains approx 60mil lines of code (incl backend - source
[http://www.informationisbeautiful.net/visualizations/million...](http://www.informationisbeautiful.net/visualizations/million-
lines-of-code/) find facebook in the list)

Nasa, budget is $19 billion. I can't find a public official record of their
code size, but the IIS is approx 6 mil lines of code, and i don't think the
other missions they did (all 33 unmanned space missions in total) are too far
off, so lets give each an average of 5mil per mission. That's 165 million
lines of code. And arguably, probably more difficult code than facebook's
domain.

~~~
evgen
And all of the code for the NASA HR system, and budget management tools, and
conference system, etc. The Facebook code base is not just the web pages you
see and backend to serve them, but encompasses just about everything used to
run the company. It is also built and maintained by a much smaller group of
coders for a much more ambiguous target domain. (And no, I do not think that
the NASA domain is in any way more difficult than Facebook's.) I guess I have
to question the relevancy of your counter-example.

~~~
coldtea
> _And no, I do not think that the NASA domain is in any way more difficult
> than Facebook 's._

You might not think it, but you'd be wrong. Facebook was for several first
years just a set of PHP crud pages and heaps of servers to run them well.

Since then they added their own compiler and such, but the core of what they
do is serving web pages at scale.

All known problems, solved by tens of teams all over the world, even at larger
scales (Google for one).

NASA's problem on the other hand is quite unique, any errors can cost lives,
and their tasks frequently include totally novel solutions for totally novel
problems related to space travel, guidance, simulations, materials science,
and so on.

Not the same difficulty at all.

~~~
jerf
"Facebook was for several first years just a set of PHP crud pages and heaps
of servers to run them well."

Personally, I think we've all got a piece of the elephant here. It is true
that there are no known methodologies that could build Facebook at the speed
it has been built without producing a globally-incoherent design. Anything
that could produce a globally-coherent design would have slowed them down so
much that they wouldn't have been such big winners in the first place. (So
"globally incoherent" isn't much of a criticism here, really.)

On the other hand, there are options other than "a big set of PHP CRUD pages
and heaps of servers" available to us now, too, and I expect those options to
continue to advance in usability. Even the various projects that improve on
PHP would bring more benefit if you could use them from day one instead of a
retrofit.

~~~
chii
Facebook didn't win because of the software, but because of marketing. At its
inception, it felt exclusive. You were special of you got an invite.

The other social networks were open season. Facebook made some good design
choices with the simple looking UI, and didn't offer the customisations
MySpace had. But that's a design choice, not software engineering choice.

~~~
eh78ssxv2f
> Facebook didn't win because of the software.

Probably true, but it's also important that they did not lose because of the
software. There are lot of great ideas that have gone in the drain because of
sloppy software.

~~~
tluyben2
Many successful companies run the worst software you will ever seen. Yesterday
there was a post about MUMPS which got me flashbacks of clients that showed me
software that ran their entire 100m+ euro / year factory mostly written in MS
Access / Excel on a shared drive (with the lovely locking Windows does!)
mostly (this particular client had some CRM and an ERP from Dutch companies
'laying on the shelf' but 'never got around to it'). One of the biggest EU
factories that creates the belts for conveyor belts runs on the worst PHP code
as ERP I have ever seen. Many sites that are not Google or Facebook etc run on
hacked together crap in whatever language and tech and do fine.

Twitter was crap at the start, most software in electronics was beyond
unusable until recently (in some systems) etc.

You can of course lose because of sloppy software, but if your marketing is
done well, I don't see why that would happen. There is an enormous
overestimation how much people care about that kind of thing here on HN. From
the first decade of the Windows versions to tons of Twitter outages after the
launch to getting their computers hacked, passwords stolen, CCs stolen,
privacy taken, slow as molasses systems, forced reboots, many crashes, bodged
updates, forced updates, virus/malware scanners taking up 50%+ of your /still
too expensive/ resources too much of the time, really bad iteration of the
Facebook app at the moment (at least on Android), broken airplane booking
forms, 404 support pages etc etc etc and yet no-one goes away because most
people curse and move on to whatever. Unless your marketing is bad aka no-one
uses it, it usually won't fail because of sloppy software.

------
sz4kerto
It's interesting to note though that they have code quality problems but that
doesn't prevent them from being massively successful.

~~~
bsaul
It's a very hard to swallow pill for engineers, but many (if not most) startup
or IT company success or failure is very independant of technological matters.

Once you get to "average" quality, then business and product decisions become
far more important than anything else.

~~~
Gracana
And it only needs to be average, or approaching average, in your niche.
There's a lot of truly awful enterprise software out there, and it gets sold
on promises of features and support, not quality.

I wish software engineering had the barriers and requirements of other
engineering fields. I deal with industrial control systems, and we have the
national electric code, UL, CE, NFPA 79, etc. With all that you can still make
a machine that doesn't work well, but at least it's going to conform to
certain conventions to make installation and maintenance easier, it will have
a robust safety system, and it will be well-documented. Most fields of
software engineering have no such minimum standards.

~~~
agentultra
> I wish software engineering had the barriers and requirements of other
> engineering fields.

It does where I live you're required to be licensed if what you're doing
passes these 3 criteria:

1\. any act of planning, designing, composing, evaluating, advising,
reporting, directing or supervising (or the managing of any such act)

2\. that requires the application of engineering principles

3\. concerns the safeguarding of life, health, property, economic interests,
the public welfare or the environment, or the managing of any such act.

This includes software development. It is illegal to use, "engineer," in your
professional title unless your accredited.

I think this is a good thing. Better software that works towards the public
interest requires liability and professional accreditation. You can't take
the, "move fast, break stuff" approach when your software is being used to
monitor and control water filtration plants or food safety processes in a
manufacturing facility. That works for the Facebook's of the world but they
are by no means the be all and end all of professional software development.

------
chiefalchemist
But seriously (i.e., ref to my previous comment). All the tools. All the
talent. All the leadership and management. All the money. And still even the
mighty FB struggles.

Point being: This shit might not be rocket scientist hard, but it ain't easy
either. And when you don't have the war chest of the likes of FB you're in for
an ongoing and never ending (quality) battle.

------
raygelogic
this strikes me the same way as the story of the guys who figured out where to
add armor to bombers during WW2. the places on the plane that had bullet holes
after a mission clearly were operational enough to return to base. the places
that never had any damage were probably critical, so that's where armor was
added.

if a company can make billions with a poor-quality codebase, clearly quality
isn't a bottom-line concern.

what is a concern is shipping the damn product.

------
jheriko
one look at hhvm tells me they have poor quality development practices. :P

i get the impression that they use the struct keyword to avoid having to type
public everywhere for instance...

------
dep_b
code quality can't handle our scale

------
p0nce
At this point it is clear that software size increase linearly with the size
of your software team, whatever the problem you solve.

This affects every organization and is something that should be actively fight
against. Having accidental, unplanned, unaccounted costs should not be the
default path.

------
strken
Can someone explain the example with the ramdisk? It's not clear to me how a
ramdisk is relevant to the problems put forward in the paper, i.e. disk is too
slow, they rely on lazy page allocation, and they have insufficient ram to
hold two copies of everything simultaneously.

------
_pmf_
> The article moves on, without wondering whether releases regularly breaking
> your app are a normal part of the software engineering process.

To be fair, that's absolutely their call to make. Nothing of value is ever
lost if your platform does not provide any non-ephemeral value.

------
peletiah
Makes me wonder how their decision for an open-plan office is related with
this...

------
tomkin
It's important to understand that Facebook is incredibly large, and the fact
that any software scaled with them to the point they are presently at, is a
"on the shoulders of giants" situation that seems to actually be working well
enough for them to become a billion dollar enterprise.

Also, the building of infrastructure that takes on scale that's not yet been
required is somewhat inefficient. But it does beg the question: If Facebook is
limited by its infrastructure, what responsibilities do they have to build
software that continues to scale for future organizations?

------
sushobhan
I don't think all the things discussed are still relevant as it's a fairly old
post. Also, "best practices" are not always best depending on the scalability
of the problem itself.

------
ChrisRus
> From Jack Reeves he cites the epiphany "that in fact the source code is a
> design document and that the construction phase is actually the use of the
> compiler and linker."

Ha. That is laughable. If it were so, then we would be able to automatically
graft features from one product to another. I contend this is exactly how SoC
are designed and there's no reason why we shouldn't copy the methods of the
digital design community (proven to scale or you wouldn't be reading this).

------
norswap
I was doing iOS dev three years ago, and the Facebook lib for iOS was the
worst piece of shit I ever had to deal with.

Their app also used to be unbearably sluggish despite not doing anything too
fancy.

~~~
babesh
I thought the joke was that they had interns write that? I remember an API
that would hold onto completion blocks and call them repeatedly instead of
just once. They would also accidentally make non backwardly compatible changes
to the API.

They didn't care since that API period propelled them to more dominance and
growth for awhile. After they shut down the viral channels, the developer
ecosystem died...but it had proven its usefulness.

The only issue now is developers don't trust them and they can't get enough
developers to build atop them.

------
baggachipz
> The “Hack” and “Move fast and break things” culture must make it very hard
> for developers to focus on quality.

Well, yeah, that's what the "break things" part means. The problem is when
people/companies try to have it both ways. "Move fast and have high quality"
isn't possible.

------
badpenny
I recommend reading They Write the Right Stuff[1] to anybody interested in
NASA's approach to writing software.

[1] [https://www.fastcompany.com/28121/they-write-right-
stuff](https://www.fastcompany.com/28121/they-write-right-stuff)

------
Bahamut
Anecdotal evidence, but I have a friend who complained that some critical
parts don't have tests, and a lot of developers are loathe to write them. He
got bit by it when he made a change in an area with no test coverage and broke
the spam filter.

------
alex-
> "when Facebook employees are not actively making changes . . . the site
> experiences higher levels of reliability."

This seems like a great thing to me. i.e. The system is stable and the error
budget is being used to facilitate change.

------
aqibgatoo
bob martin a.k.a uncle bob always says that people consider patterns like
mvvm,mvc,mvp etc as Architecture which actually are just for the ui layer.It
happens commonly in mobile development.

------
pablo_fiesta
Rub lavender oil on the servers, say "web-scale" 3 times, click you heals and
everything will be OK. Oh wait, but the Russians.

------
jyriand
Only Kent Beck can save them. :)

------
typetypetype
What's the point of an app? Generating high quality code or generating
revenue? Until you hit intersections where revenue can be increased by
changing code, the priority will always be on new or changed features.

------
chiefalchemist
Move fast...and just leave things broken. Users be damned.

------
dwils098
links to slides are all dead... in parent articles and reference articles

------
linkmotif
While the author wrote this article FB made millions of dollars.

------
neals
[2015]

------
_pmf_
If a company hails React as a sane way to develop software, you know where it
stands.

~~~
peletiah
Just trolling or could you argue that?

