
Notes from Facebook's Developer Infrastructure at Scale F8 Talk - cpeterso
http://gregoryszorc.com/blog/2015/03/28/notes-from-facebook%27s-developer-infrastructure-at-scale-f8-talk/
======
necubi
The most interesting part for me was

    
    
        > They appear to use code coverage to determine what tests  
        > to run. "We're not going to run a test unless your diff 
        > might actually have broken it."
    

This seems like such an obvious optimization in hindsight, I'm surprised it's
not more common. Does anybody know of other CIs that can do this?

~~~
dperfect
I'd be hesitant to rely 100% on this approach (and from the notes it sounds
like FB still runs the other tests - just less frequently in some cases),
especially where the code coverage tools aren't guaranteed to account for
unintended interactions between seemingly unrelated pieces of code. In theory,
well-written code/tests shouldn't have those kinds of problems in the first
place, but larger projects often don't deserve that level of trust (in my
opinion).

Still, I think it's a great idea for giving quick, "preliminary" feedback to
developers - as long as the full test suite is still run periodically.

~~~
necubi
Yeah, absolutely. You'd want to run your full test suite (which probably
includes heavier integration tests as well) before deploying, but I think in a
lot of cases it's valuable to provide, say, 95% accurate feedback quickly.

~~~
icefox
Another way to put it would be I will happily only take 95% if the other
option is we don't do anything and people can even commit stuff that doesn't
even build let alone pass tests.

------
larsberg
This talk and my experiences at Microsoft are why I tend to try to steer our
projects at Mozilla Research to partner with external offerings rather than
doing too much building of our own. It's hard to imagine a world where we
build _and continue to improve_ a source control / issue tracking system
better than GitHub; a continuous integration solution better than Travis CI; a
build system; etc.

Even if we can imagine building a better product today, the ongoing
maintenance and extension of those services quickly become full-time multi-
person investments with many hundreds of dedicated machines, neither of which
even our core infra teams have, much less the little tiny research org.

~~~
bzbarsky
It's not hard to imagine a world in which someone builds or continues to
improve an issue tracking system better than GitHub, because GitHub seems to
be actively uninterested in improving its issue tracking system in various
dimensions that matter to a lot of issue tracking system users (ability to
attach testcases, dependency tracking, and release management are the big ones
for me personally). It's not just that GitHub is lacking in those areas;
they've repeatedly rejected suggestions for improving them.

~~~
larsberg
> they've repeatedly rejected suggestions for improving them.

I certainly agree that it's easy to imagine better issue management or patch
review than GitHub has today - like many other projects we use other systems.
When I've talked with GitHub engineers, all of our issues are well understood
and they have plans to address them.

My skepticism is around the strategy of building and supporting our own things
vs. partnering with places like GitHub that make a living building and
supporting such software. In the Mozilla-specific case, they have proven open
to working with us in the past, by pony-trading enhancements to Firefox for
enhancements to GitHub, and the inability to deliver has been more on our side
:-/

~~~
bzbarsky
Interesting. I haven't seen any requests for enhancements to Firefox (or more
precisely Gecko) from GitHub, and I watch all incoming Gecko bug reports...
Were they asking for these enhancements from Firefox itself, not Gecko, or
through some sort of opaque channels or something?

Past that, partnering with people makes sense to me if they're flexible enough
to add things when we need them. My point is simply that historically GitHub
has not been adding the things we need.

------
beliu
On the topic of having one IDE that just works for multiple
languages/platforms, you should check out srclib
([https://srclib.org](https://srclib.org)), an open-source language analysis
tool (I'm one of the creators) that abstracts away the language-specific parts
of IDE support so that IDEs/editor plugins can hook into one API and
automatically get support for a bunch of different languages. This way, you
don't have to build MxN plugins (M editors, N languages) to support all combos
of editors/languages people might want to use.

The currently supported languages are Go, Java, Python, JavaScript, and Ruby,
so this wouldn't work yet for iOS, but Objective-C and Swift are high on our
priority list. Editor plugins exist for Emacs, Sublime, and Atom. Would love
to hear anyone's feedback from trying it out.

~~~
wting
This sounds a lot like Steve Yegge's Grok:

[http://bsumm.net/2012/08/11/steve-yegge-and-
grok.html](http://bsumm.net/2012/08/11/steve-yegge-and-grok.html)

It's unlikely that his version will ever be open sourced or usable outside of
Google though.

~~~
durin42
IIRC [http://kythe.io](http://kythe.io) is at least related by heritage to
grok.

~~~
beliu
Yep, Kythe is the open-source version of grok. It's awesome, and the srclib
team is in touch with the Kythe team at Google to build on top of their
analysis libs (for C++ and Java).

Long term vision is to get rid of the "my editor X doesn't support language Y"
problem as well as build a lot of other multi-language analysis tools on top
of it. Shameless plug: if you care about developer productivity, join us!

------
rurban
"We made mercurial up to 50x faster than git, more than 2000 improvements" on
big repos with lot of history.

This would be their blog post about this:
[https://code.facebook.com/posts/218678814984400/scaling-
merc...](https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-
facebook/) and the code at
[http://selenic.com/repo/hg/shortlog/](http://selenic.com/repo/hg/shortlog/)

------
raverbashing
Interesting info about XCode and the FB app. I wonder if Apple is
collaborating with FB on this or if it's mostly deaf ears...

"So they built their own IDE. Set of plugins on top of ATOM. Not a fork. They
like hackable and web-y nature of ATOM.

The demo showing iOS development looks very nice! "

Seems that they have the right idea about "optimizing for Developers"

~~~
coldcode
Although I don't really like XCode but use it all the time, what the hell
takes 5 minutes to open a project????? The FB app isn't that big. Maybe it
includes downloading an entire repo or something? I've worked on big iOS apps
and this was never an issue.

~~~
insaneirish
Facebook has a 54 GB monolithic repo (as of almost a year ago, April 2014).

Primary source:
[https://twitter.com/feross/status/459259593630433280](https://twitter.com/feross/status/459259593630433280)

HN article:
[https://news.ycombinator.com/item?id=7648237](https://news.ycombinator.com/item?id=7648237)

------
Bahamut
A lot of nice summaries and points, but one caught my eye - the mention about
building Nuclide on top of Atom. My experience with Atom is that it gets
painfully slow quickly, with me experiencing a crash every few days.

I would like to hear more thoughts from FB folks on their experiences.

~~~
SwellJoe
Java desktop applications have shown that with enough resources thrown at the
problem, slow and flaky things can become fast and stable. JavaScript desktop
applications seem to be at the "slow and flaky but has tons of resources being
thrown at them" stage.

The VM, as with Java ten years ago, is also seeing remarkable improvements.

------
freedrull
"Push-pull-rebase bottleneck: if you rebase and push and someone beats you to
it, you have to pull, rebase, and try again. This gets worse as commit rate
increases and people do needless legwork. Facebook has moved to server-side
rebasing on push to mostly eliminate this pain point. (This is part of a
still-experimental feature in Mercurial, which should hopefully lose its
experimental flag soon.)"

What if there are merge conflicts? I don't know about Mercurial, but in Git
there are tons of cases where rebasing cannot happen automatically.

~~~
evgen
The process is optimistic. You submit your commit and it runs async in the
background with success or failure sent to you via email and SMS. You still
run into merge conflicts, especially if you are touching a frequently tweaked
bit of core, but not having to babysit the process is almost always a win.

------
coldcode
Although I love reading how big companies solve developmental problems I am
also glad I don't work there. Programming at FB seems more like working in a
salt mine.

------
ghuntley
Does anyone have a .torrent of all the videos from F8? It's a shame that they
are self-hosting otherwise it would be possible to youtube-dl it :-)

~~~
jamesgpearce
The engineering sessions are in this playlist:
[https://www.youtube.com/playlist?list=PLb0IAmt7-GS1_7FcSupSJ...](https://www.youtube.com/playlist?list=PLb0IAmt7-GS1_7FcSupSJoUe21tF12eu8)

~~~
ghuntley
Thankyou.

------
brazzlemobile
Anyone have a mirror? Getting gateway timeouts.

~~~
indygreg2
The site is hosted via GitHub pages. Maybe it's not available to you due to
the lingering GitHub DoS. If you care to read HTML and can manage to get
through to GitHub:
[https://github.com/indygreg/indygreg.github.com/blob/master/...](https://github.com/indygreg/indygreg.github.com/blob/master/blog/2015/03/28/notes-
from-facebook%27s-developer-infrastructure-at-scale-f8-talk/index.html)

------
131hn
* Yep, another "better than ever"/"last one you'll ever need" IDE. Thank you facebook. (see you soon for the next one)

* A DVCS that rely on a central server for merging (sandcastle) is no longer.. distributed... (and you cannot have distributed team work here, this is wrong in multiple way)

* I think i'll never let a centralized, monolithic repository to be set up in my company. All great stuffs/talent i ever learned came from differents sources, differents independant projets (from git, hg or SVN). Loose that and i think i'll get narrow minded.

* All those fancy stuffs make facebookers better "facebook developers" but less prone to share with others (we don't share the same language, culture or tools anymore, even the design of the monolithic repo cannot help here)

* This is more a lesson on "how we made X thousands lambda devs work together" than actual tools i might need to use someday.

* I'll bet that facebook "infra" developers are less likely to use the tools they describe, and that is ironic. Those tools mostly apply to some hidden mass (100k commit / week.. ) of uniform developers that I'll never meet.

~~~
DannyBee
"A DVCS that rely on a central server for merging (sandcastle) is no longer..
distributed... (and you cannot have distributed team work here, this is wrong
in multiple way)"

Yet doing things like rebasing or merging pull requests properly, requires you
have an up-to-date master, making it a centralized operation.

People seem quite happy with this.

~~~
hueving
>Yet doing things like rebasing or merging pull requests properly

Rebasing is a smell of bad source code management. Merging pull requests is
fine whether or not you are up to date. I don't know what you mean by
'properly'.

~~~
indygreg2
It is much easier to bisect linear history than history with merges.

~~~
hueving
How is it any different? I do it all of the time with no problems.

