
Static Analysis at Scale: An Instagram Story - YoavShapira
https://instagram-engineering.com/static-analysis-at-scale-an-instagram-story-8f498ab71a0c
======
joatmon-snoo
A third static analyzer for Python! I wonder what this landscape is going to
look like in a few years... (There are probably others, but this is the first
I'd heard of Pyre, and I'd only known about mypy and pytype up until today -
if there are others I'd love to hear about them!)

On a more substantive note, static analysis is one of those things that sounds
like you shouldn't even need it but in reality is a surprisingly powerful
tool, for the same reason that all code has bugs: humans honestly suck at
writing code. (For those of you that have seen Bret Victor's "Inventing on
Principle", I feel like static analysis is one of the first steps in getting
to the kind of feedback loop he envisions.)

We have all these ideas around how things _should_ work, but so frequently our
mental model diverges slightly from what's actually happening (which dynamic
typing only makes worse, and NPEs even more so) and that's on _top_ of the
mistakes we make (like this one in the JDK, which was caught using ErrorProne,
a static analyzer for Java:
[https://bugs.openjdk.java.net/browse/JDK-8176402](https://bugs.openjdk.java.net/browse/JDK-8176402)).

Also, a fun tidbit about using static analysis to apply automated code fixes:
this is basically when you realize you _need_ auto-formatters, because you
still want your code to be indented like a sane person would after applying a
change like this. (And imagine how much more complex the
autoformatter/autofixer have to be when you think about how they have to do
things like preserve comments, etc!)

~~~
schwag09
_if there are others I 'd love to hear about them!_

I've been collaborating with Duo Security to build a new Python static
analysis tool that focuses on security deficiencies: [https://github.com/duo-
labs/dlint](https://github.com/duo-labs/dlint)

In general, I agree, static analysis is a very powerful technique. I'd like
the computer to double-check my work as thoroughly as possible when I'm
working with code. Static analysis tools are often very fast and essentially
"free" to run, so why not? False positives often become the limiting factor,
but in my experience they at least point to locations in code that someone has
deemed noteworthy, and perhaps should be investigated. Squelching false
positives is also typically an easy process.

Whether it's simple stylistic recommendations for code consistency, security
best practices, or even disseminating codebase information (e.g. function
deprecation notices from the Instagram article), static analysis is a very
useful technique.

------
aidos
Interesting article. I’d love to hear more about how other folks are getting
on with typing in python these days. What’s the preferred tool for limiting
etc? I’ve been playing with Microsoft python language server recently and mypy
(via vim coc). I haven’t got a config that works well yet but it shows promise
and I’m spurred on by how typescript is working well for us in that setup.

~~~
lord_braleigh
Instagram is building a tool called Pyre ([https://pyre-
check.org/](https://pyre-check.org/)).

------
curiousgal
> we have hundreds of engineers shipping hundreds of commits every day

Can someone with (any) experience explain to me why do seemingly perfectly
functional websites need change all the time? Is the production version hacked
together or what? Why can't websites be coded once and left to run with the
rest of the effort being devoted to maintance/adding more servers as the load
increases?

I admit that I know almost nothing about how large codebases function (it
might being apparent judging by the question)

~~~
mmq
The changes are not only about making the website/app stable, it's mostly
about shipping new features and product ideas.

The space where Instagram operates is very competitive, so they need to keep
innovating and exploring new ideas to grow the product, increase usage, and
improve retention...

------
antpls
I quickly looked at pyre and pytype. I wonder : is there a static analyzer for
Python that can check for termination guarantee in small snippets of code?

The idea is to have a "safe" and "unsafe" Python (ala Rust) according to some
guarantees. Proving termination of program is hard, but it is doable if dev
time is dedicated to it, and algorithms are well chosen.

That way, some core libraries could be rewritten and give stronger guarantees.

The idea is used in [https://github.com/google/starlark-
go](https://github.com/google/starlark-go) which is a subset of Python. People
who are used with Python can use it to write imperative config files, and the
files are guaranteed to not mess with sensible stuff

edit : some starlark design choices explained :
[https://github.com/bazelbuild/starlark/blob/master/design.md](https://github.com/bazelbuild/starlark/blob/master/design.md)

------
allsunny
Would be cool if they could quantity the gains in productivity - eg before we
put in the static analysis we had x level of software defects, following these
improvements we saw a drop to level y. Trying to introduce similar
improvements at my current place but it’s a hard sell without data

~~~
joatmon-snoo
The coverity paper may be interesting to you:
[http://delivery.acm.org/10.1145/1650000/1646374/p66-bessey.p...](http://delivery.acm.org/10.1145/1650000/1646374/p66-bessey.pdf?ip=104.132.11.86&id=1646374&acc=OPEN&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E6D218144511F3437&__acm__=1565929701_4dc880fcf10483a0346296d0efb311c2)

~~~
merlinsbrain
The link doesn’t work, I’m interested to read it though.

~~~
mdibiase
Try this link. It should be the same article as linked previously.
[http://www.cs.columbia.edu/~junfeng/14fa-e6121/papers/coveri...](http://www.cs.columbia.edu/~junfeng/14fa-e6121/papers/coverity.pdf)

------
protomolecule
I wonder if there was a point in the history of Instagram's codebase at which
it would've been cost-effective to rewrite it in a statically typed language.

If you find yourself in a hole, stop digging.

~~~
dominotw
how would static typing help with "method fn is deprecated use foo' linting ?

~~~
protomolecule
Quoting the article: "The benefits of better typed code are obvious, but leads
to yet another benefit: Having a fully-typed codebase unlocks even more
advanced codemods."

------
yakshaving_jgt
It's curious to me that commonly in programmer discussion, we'll say on the
one hand that the programming language you use doesn't matter, while on the
other hand tout the benefits of adding types and static analysis.

There _are_ some languages where types and static analysis are _part of the
language_.

It's paradoxical that we as an industry hold these two things to be both equal
and different.

~~~
mattip
Citation needed. I recall people saying they prefer the concise syntax and
speed of development with interpreted languages that do not force type
annotations and others who prefer the runtime speed and compiler warnings in a
typed language. I don’t recall anyone saying they are interchangeably
equivalent.

~~~
yakshaving_jgt
\- [https://medium.com/@schamne/why-your-choice-of-
programming-l...](https://medium.com/@schamne/why-your-choice-of-programming-
language-doesnt-matter-b15f2abb0b69)

\- [https://www.codementor.io/agustinchiappeberrini/why-the-
it-d...](https://www.codementor.io/agustinchiappeberrini/why-the-it-doesn-t-
matter-what-programming-language-you-learn-aih53kaia)

\- [https://www.quora.com/If-truly-programming-languages-dont-
ma...](https://www.quora.com/If-truly-programming-languages-dont-matter-how-
come-most-of-machine-learning-development-today-is-done-in-Python-and-not-
some-functional-languages-like-Haskell)

\- [https://spin.atomicobject.com/2015/07/15/language-doesnt-
mat...](https://spin.atomicobject.com/2015/07/15/language-doesnt-matter/)

\-
[https://learnpythonthehardway.org/book/advice.html](https://learnpythonthehardway.org/book/advice.html)

The idea that programming language does not matter is a pretty commonly-held
view.

------
dominotw
> Lets say we needed to deprecate a function named ‘fn’ for a better named
> function called called ‘add’.

wondering why they would deprecate fn instead of renaming everywhere with
'add'. I thought that was one of the main big sells of the monolith.

~~~
joatmon-snoo
At a certain scale, it becomes prohibitively difficult to rename everything at
once, for a few reasons:

* Hyrum's Law - people do terrible things, and a change that seems safe almost never is. Doesn't matter what people _should_ do, if they can they'll do it. Having good tests and scalable CI is the only guard for this.

* Moving a target is hard. If you try to change a moving target A into B, and you're changing a couple thousand references to A, chances are pretty good that while you're getting your A->B change reviewed, someone will add more code that depends on A, and then you can't submit your A->B change without fixing the new reference. The solution here is you check in B, tell people to use B instead of A (which checks the growth of references to A), make A point at B, maybe as an implementation detail (which causes all references to A to depend on B), and incrementally change indirect dependencies on B through A to be direct dependencies on B.

* Depending on a moving target is annoying. If N people write code depending on A, and a commit goes in changing A->B, then that's N people whose productivity you've hurt. In practice this one isn't really that bad, just something to think about.

* Small changes are more easy to roll forward than large ones. This assumes your change is rolled back, but consider this scenario: your change A->B, unbeknownst to you, tickles some non-deterministic behavior (maybe a race condition) when used in a specific way, and your CI doesn't fail when you submit the change at EOD. You come in tomorrow morning and find out that the change was rolled back, because it caused flakiness for N tests went from 0.1% to 25%. If your change was a small, targeted one, it'll be much easier to trace down the non-determinism and understand how it caused this change; but if it was a big one, not only might it be harder to trace down the non-determinism, there might even be other similar non-deterministic bugs that your change is causing. All of these issues will conspire to make it harder for you to make progress on your large-scale change.

When you're changing 10 references, maybe even 20-30 or so (and for some
changes this number can go even higher, e.g. renaming an internal Java
package), it definitely makes sense to do an A->B type change in a single
commit. (And in this situation, the monolith does come in handy, because you
don't have to wait for a version bump to propagate the change.) But at O(1K)
LOC, this isn't a super tenable.

