
Python at Scale: Strict Modules - apadmarao
https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834
======
timothycrosley
More and more I want someone to create a new language that amounts to a strict
subset of Python, with mypy built-in, and is compilable into machine code.
Python has by far my favorite syntax, community, and in my experience leads to
the greatest productivity. There just happens to be a lot of overly dynamic
features, that aren't even used by most, but used just enough to hold back
optimization and structural improvement.

~~~
cpeterso
I like Python, but I often wonder how many developers use Python because they
actually use dynamic language features versus just liking the languages' clean
syntax and library ecosystem. I'm surprised languages that offer both REPL
(for development) and AOT native compilation (for production), like OCaml, are
not more popular. Evidence that syntax matters, I guess. :)

mypy and mypyc are interesting but their compile-time checks and optimizations
are still hampered by Python's dynamic language semantics.

~~~
clintonb
Don’t underestimate inertia. I’ve worked with Python and Django for seven
years. I know the libraries in the ecosystem. I know the framework. It’s far
easier for me to start a project with Django than to learn another framework
or language.

------
allan_s
> This means that just by importing this module, we're mutating global state
> somewhere else.

Yes, this !

That's why I hate Django and some flask app the most for, the fact that by
importing a module, you're implicitly creating a database connection, and a
lot of other magic stuff, which mean that now I can't import a constant
defined in said module outside of `python manage.py`

Also as said below in the article, suddenly it's much harder to handle
smoothly the "the database is momentary unavailable" (because someone has put
the line starting the database connection in the global space of a module
somewhere)

I much prefer frameworks/modules for which code is executed only once you
invoke their "setup" function

~~~
nerdponx
_I much prefer frameworks /modules for which code is executed only once you
invoke their "setup" function_

Django _does_ have a "setup" function. You can't import and use Django
database connections outside of a running application without it.

Flask also has a "run" method and does no i/o without it.

~~~
heavenlyblue
Every time I hear a comment alike parent's, it makes me think how many times a
day I actually read a comment in the same fashion, but about something I
actually know nothing about.

~~~
nerdponx
With credit to the original poster, they might be complaining about the fact
that Django is a monolithic framework and you can't really use Django code
_without_ spinning up the i/o portion. Which is legitimate criticism, but
frankly if that's what you need then you shouldn't be using Django.

~~~
heavenlyblue
If by I/O you mean web i/o then Django provides a perfectly functional
management utility interface accessible through cli.

If, however by i/o you mean the database portion of it then Django works
without database configuration, too.

------
ledauphin
I love the idea, but it feels like just an idea at this point. I'd rather read
about them releasing their 'compile-time' analyzer and revealing their
measurements for how much startup time it saves.

In our codebase, we have pretty strict developer-enforced rules about not
doing I/O at the module level, usually through the use of simple "Lazy"
wrappers for module-level objects. I'd be curious to know what other
approaches people have taken with Python here.

~~~
rectangletangle
It is an interesting approach, though I feel like this could introduce some
nasty unintended consequences given how dynamic and introspective Python can
be (admittedly I haven't studied this particular implementation).

I always treated this a bit like single underscore private functions/methods,
i.e., follow a convention that produces code that's easy to reason about, even
if it's not strictly enforced by the language/compiler. So in practice this
equates to separating out modules that mutate global state, and placing the
majority of logic in "strict" modules that only declare a bunch of "pure"
classes/routines. So the "non strict" code is really just a thin layer of
wiring gluing everything together. For instance my Celery task files tend to
be very thin.

~~~
ledauphin
well, we also heavily use static typing, so you end up with something like

my_db_conn: Lazy[DbConn] = Lazy(lambda: make_db_conn(...))

and MyPy will tell you if you're doing something silly when you try to use it.

EDIT: After typing up this response and submitting I realize you were talking
about their strict approach rather than ours. whoops :)

------
ben509
> How do we know that the log_to_network or route functions are not safe to
> call at module level? We assume that anything imported from a non-strict
> module is unsafe, except for certain standard library functions that are
> known safe.

It's hard to know anything about the stdlib as it can be monkey patched, e.g.
[1]

That said, you could solve this with diagnostics; calculate signatures of
stdlib functions and classes to find any known safe ones that were patched.
Run that check in your test suite to find problematic imports.

> If the utils module is strict, then we’d rely on the analysis of that module
> to tell us in turn whether log_to_network is safe.

I like this. It seems far more usable than proposals like adding const
decorators.[2]

[1]:
[https://github.com/gevent/gevent/blob/master/src/gevent/monk...](https://github.com/gevent/gevent/blob/master/src/gevent/monkey.py)

[2]:
[https://github.com/python/typing/issues/242](https://github.com/python/typing/issues/242)

------
jedberg
It's interesting to me that they are going down this path instead of the
microservices path. This seems like something ripe for slowly breaking down
into microservices.

Someone made a change that took down production because of non-deterministic
outcomes? How about break out whatever they were changing into it's own
service? With proper fallbacks, breaking that part shouldn't take down all of
production again.

To be clear, I'm not saying microservices will solve all their problems or be
less work. I'm just saying that with an equal level of effort, they would
probably get more overall reliability by having multiple services, they'd be
able to use multiple languages, whatever is suited to the task at hand, be
able to deploy even more often with less risk, and be able to isolate these
types of "change on import" behavior to a much smaller surface on any given
deployment.

~~~
coldtea
> _Someone made a change that took down production because of non-
> deterministic outcomes? How about break out whatever they were changing into
> it 's own service? With proper fallbacks, breaking that part shouldn't take
> down all of production again._

Yeah, now you'll have 10 interconnected services, 10x the complexity, and
everything will have the ability to take down all of large parts of
production, plus all the extra pain points of a distributed system...

~~~
jedberg
You won't have 10 times the complexity if you are taking a monolith and making
each section services. You'll have to same dependency graph, it will just use
the network to make calls between them instead of being local.

You'll have added complexity with the network calls, which is why I said it
wouldn't be any less work, just different work.

~~~
coldtea
> _You won 't have 10 times the complexity if you are taking a monolith and
> making each section services. You'll have to same dependency graph, it will
> just use the network to make calls between them instead of being local._

Merely "use the network to make calls between them instead of being local"
will add 10 times the complexity -- you suddenly have a distributed system,
latency, delays, parts that can be on or off, de-centralized configuration
(which can also get out of sync), and so on.

------
miki123211
This is yet another example of the divide between wizarding and
engineering[1]. When you're a small startup, what matters is the
expressiveness of your language, and the ability do do a lot of things very
very quickly. Type safety, performance, readability, those things don't
matter. You're just a bunch of engineers who know the whole codebase inside
out, you're pretty certain of what you're doing. In short, you're wizarding.
If you grow big enough, this approach slows you down greatly, and you need to
switch to engineering. You sacrifice some speed for making the codebase more
understandable to a larger group of people, you can no longer assume everyone
knows all the code, you write unit tests, need types and dislike
metaprogramming because of the confusion it creates. This is why languages
like Python, Ruby, Lisp or Smalltalk are amazing for small startups, but Java
is what enterprises use. They're different ends of the wizarding/engineering
spectrum. I wish there was a language that let you move gradually from one end
to the other, exactly when you need to.

[1] [https://www.tedinski.com/2018/03/20/wizarding-vs-
engineering...](https://www.tedinski.com/2018/03/20/wizarding-vs-
engineering.html)

~~~
TylerE
Who is starting large-scale new projects in Java in 2019?

~~~
typon
It's a good question. Why would you pick Java over Go or C++?

~~~
WatchDog
Why would you pick Go or C++ over Java?

C++ is a hydra of complexity, sure it has it's place, but it's not nearly as
productive as Java for your typical web application.

Go is almost the opposite, so simple it lacks features like generics. The last
time I used Go it had fundamental usability issues around dependency
management(although I think recent versions have improved on vendoring a
little).

~~~
adev_
> C++ is a hydra of complexity, sure it has it's place, but it's not nearly as
> productive as Java for your typical web application.

Modern C++ well is as productive ( probably even more productive ) as Java.
The main issue with C++ is recruitment, C++ engineers are rare because C++ is
barely teached.

~~~
rswail
C++ is barely taught in ProgrammerGenerationFactories because "modern" C++
still allows "old" C++ and makes it difficult to stop developers from doing
that.

Just like MISRA tries to constrain C programmers from doing dumb things in the
embedded world, "modern" C++ tries to the same in the business world. But
there isn't an easy way to enforce it, especially when you're outsourcing to
some code sweatshop.

~~~
adev_
> But there isn't an easy way to enforce it

Every mature enough language has a subset that you need to avoid. Including
Java. This is precisely due to this kind of things that every company need to
have coding guidelines and proper static analysis tools.

> especially when you're outsourcing to some code sweatshop.

If you outsource your dev to cheap, other side of the world, low quality
engineers. Then you deserve your problem, in any language.

I worked in the past for a company (embedded programming) that had an entire
team of expensive engineers in Luxembourg just to fix the stupidities of an
other team of outsourced engineers in India.

------
k_sze
Another thing that I would like to see in some kind of strict mode is the
ability to mark explicit exports like in JavaScript modules. I often want to
import multiple things globally at the top of a module because they are shared
by multiple class or function definitions that I am writing. However, such
imports end up being exposed to and usable by the consumers of my module, even
though the consumers should really have imported those things at their source
instead of via my module.

There are currently maybe two ways to tackle this “problem”, without a strict
mode:

1\. Don’t import at the global module scope; but that’s a bit tedious.

2\. Import with rename, like `import os as _os`, and then leave it to the
principle of “we’re all consenting adults”. I.e. if anybody imports and used
things that start with an underscore, it’s clearly their fault, not mine.

~~~
andreareina
3\. Import as normal, and leave it to the principle of "we're all consenting
adults"; unless something is explicitly called out as being part of the public
API I consider Law of Demeter[1] "violation" the same as accessing _var.

[1]
[https://en.wikipedia.org/wiki/Law_of_Demeter](https://en.wikipedia.org/wiki/Law_of_Demeter)

------
alexchamberlain
I think this is an interesting idea, which appears to embed a stricter subset
of Python within Python itself. Have the Instagram engineers tried floating
this with the wider community via established channels like Python-Ideas or
discuss.python.org?

------
jbmsf
I like the idea, but it feels a bit heavy handed outside of a very large team.

I think the first step here is to get away from the assumption that importing
a module will have "interesting" side effects. This is not only a problem with
Python...

I tend to create mini "dependency injection" frameworks that create a pattern
for loading module code at some point well after import. This patterns tends
to reduce to wrapping whatever code you have in the module in a
function/closure instead of just running whenever.

Again, I like the idea of enforcing constraints with code, but I don't think
it's a substitute for educating developers to avoid certain patterns and
giving them infrastructure that makes the alternative easy.

------
marcoseliziario
[https://docs.python.org/3/library/importlib.html#importlib.u...](https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader)

------
time4tea
Wow. Talk about solving the wrong problem!

Millions of lines of code in a monolith. 20s start up time. Meta monkey
patching. One unit test per process... Yikes!

Software architecture, anyone?

Maybe Instagram should get a copy of Michael Feathers' book...

------
rurban
I like that idea, it's just not that easy. How to do define module versions
and inheritance, when you are not allowed to do global assignments in the
module. declarations only, and no IO or global side effect is fine, but
declaring versions and inheritance need to be allowed in global scope.

I added these ideas here:
[https://github.com/perl11/cperl/issues/406](https://github.com/perl11/cperl/issues/406)

------
tahdig
> ... many of whom are new to Python.

well, if you ask me to write language X, I would definitely make mistakes for
the first couple of weeks/months/years, that is why you need code review,
mentoring and education plans for your hires.

> Here’s another thing we often find developers doing at import time: fetching
> configuration from a network configuration source.
    
    
      MY_CONFIG = get_config_from_network_service()
    

I am pretty sure this an anti-pattern, if this code passed the code review,
you should make your review process more strict.

    
    
      def myview(request):
        SomeClass.id = request.GET.get("id")
    

> Likely you’ve already spotted the problem

Well, yes, why would you do this? why would this pass code review? why do we
we have linters and other checks for dynamic languages

> It works great for smaller teams on smaller codebases that can maintain good
> discipline around how to use it, and we should switch to a less dynamic
> language.

It seems we are here blaming python for shortcomings of a monolith also,
instead of chunking out specific businesses modules to separate
services/micro-services.

TO be honest the strict mode seems interesting, but I believe the problems
they seem to be facing can be solved by a couple of changes to their pocess
and code:

\- everyone gets a mentor if they are not experienced in python or django

\- code review atleast by two experienced python developers(does not count if
you have coded for Java for 20 years)

\- teams should try to move their logic outside the monolith(it sounds like
they have a monolith)

\- write CI tests to measure how much time it takes to import a file, if it
takes more than T(line count * LINE_PROCESSING_THRESHOLD) you have to fix your
code.

\- prepare config and load it before running the actual server, no network
call for getting config

All in all, python is suitable for big companies also, the thing is if don't
care about the best practices, you would also have problems when you are a
small startup, but in a big co it would make it impossible to move forward,
trick is to independent of the company size follow best practices and have
code review.

~~~
scrollaway
That's a long post to say "do more code review instead of investing into
technical solutions to technical problems".

Clearly, Instagram's solution saves them time. That means faster code reviews
which incidentally makes them more accurate. Your post doesn't really make
sense.

------
avip
It's very important to think about objects lifecycle management.

It's also important to... use pytest fixtures instead of arbitrarily patching
around in tests.

------
konschubert
I have a question about a detail in the article:

> But if we moved the log_to_network call out into the outer log_calls
> function, [...] this would no longer compile as a strict module.

My current understanding is that the log_calls method would NOT get executed
during module load time!?!

Why would having a side effect in this function violate the intention of
__strict__ ?

~~~
scrollaway
> _My current understanding is that the log_calls method would NOT get
> executed during module load time!?!_

That's incorrect. log_calls gets executed on import because it's a decorator,
so equivalent to `hello_world = log_calls(hello_world)` at the top-level
(which does also get executed).

log_to_network in the _wrapped() definition doesn't get executed until
hello_world gets called; but outside of the definition of _wrapped does get
executed.

~~~
konschubert
Right! I missed the fact that log_calls is used as a decorator further down.

------
tln
Avoiding module side effects and making classes and modules immutable seem
like two separate concerns

~~~
bjoli
Not really. Mutation in general, and in modules in particular, inhibit a lot
of reasoning about the code, and thus stops a whole lot of optimizations from
being possible. Guile (a scheme dialect) recently got declarative modules for
that reason, where a top level binding cannot change (i.e. you cannot set! a
binding, but you can wrap in it a mutable container and change the contents of
that container). This makes procedure calls and variable lookup a lot faster.
Andy Wingo wrote about it here:
[https://wingolog.org/archives/2019/06/26/fibs-lies-and-
bench...](https://wingolog.org/archives/2019/06/26/fibs-lies-and-benchmarks) .

Those optimizations won't mean much for cpython, since Cpython doesn't try to
run things fast, but for something like pypy this could be a big deal.

~~~
bjoli
To quote the article (from.memory): "adding static modules is probably the
single most important optimization guile can do in the near future".

The quote is probably wrong, but it is right in spirit.

------
ianamartin
Try Zope.Interface and Pyramid for a framework. You'll be really happy.

------
accidentaldev
who would have thought Instagram is a python monolith. ?

------
carapace
> Instagram Server is a several-million-line Python monolith

That's bananas.

Nothing Instagram does requires that much code.

Also, that much Python code means you're doing it wrong.

~~~
carapace
No, I'm seriously you guys.

Python is too expressive to require mega-LoC for that site.

You could implement an OS, relational DB, spreadsheet, and optimizing compiler
all in less than that.

~~~
orf
You have no idea about their codebase, the implementation details of their
features nor how they counted the lines (comments included?). So stating that
it’s dumb is beyond ridiculous.

You are right in that it’s certainly a high LoC count for Python, but still...

~~~
carapace
I didn't say "dumb" I said "bananas".

And yes, knowing nothing else about their code base than A) It's in Python,
and B) it's several million lines of code, I feel very confident that there is
at least an order of magnitude too much of it. Instagram is just not doing
anything that complicated.

(I should mention I specialize in maintaining and refactoring legacy Python
code. I know what I'm talking about here.)

~~~
ClippyIO
Features that are "not complicated" can actually very easily be "very
complicated" at scale. Which Instagram does have. 500 million users, every
single day.

~~~
depressedpanda
Fewer LOC would actually benefit them at scale.

If you need _several millions_ of lines of Python to do what Instagram server
does, the code is bloated.

My bet is that they let too many Java devs loose on the code base, without
experienced Python devs reviewing the commits and managing the deluge of
unnecessary classes. I've seen it happen before.

~~~
pytester
>If you need several millions of lines of Python to do what Instagram server
does

I have this feeling that you're probably not all that aware of 95% of what
their code actually does, and thus probably not in a position to make
judgements as to whether their code base is truly bloated relative to what it
does.

------
zestyping
I like this a lot.

------
zallarak
This article is among the best argument for using a typed language I’ve yet
seen.

~~~
kbd
This has nothing to do with types. It's more about static guarantees the
language gives about module import behavior.

~~~
nothrabannosir
In OP's defence:

 _> So that's a third pain point for us. Mutable global state is not merely
available in Python, it's underfoot everywhere you look: every module, every
class, every list or dictionary or set attached to a module or class, every
singleton object created at module level. It requires discipline and some
Python expertise to avoid accidentally polluting global state at runtime of
your program._

 _> One reasonable take might be that we’re stretching Python beyond what it
was intended for. It works great for smaller teams on smaller codebases that
can maintain good discipline around how to use it, and we should switch to a
less dynamic language._

 _> But we’re past the point of codebase size where a rewrite is even
feasible. And more importantly, despite these pain points, there’s a lot more
that we like about Python, and overall our developers enjoy working in Python.
So it’s up to us to figure out how we can make Python work at this scale, and
continue to work as we grow._

Those are literal quotes from the article. That is quite damning. How did they
get to this point? By starting when Python was appropriate, and taking it day
by day.

~~~
kbd
How is "our developers really like Python even on a million-line codebase,
despite its global mutable state requiring discipline" "quite damning"?

------
brenden2
It still blows my mind that people don't use strongly typed languages in the
first place and spare themselves from all this future pain.

My guess (based on my experiences) is that companies wind up in this position
from having inexperienced people building early versions of products instead
of hiring experienced engineers (who are usually more expensive).

~~~
b3orn
Python is strongly typed, just not static.

~~~
brenden2
Python uses duck typing:
[https://en.wikipedia.org/wiki/Duck_typing](https://en.wikipedia.org/wiki/Duck_typing)

I would categorize it as a subset of dynamic typing, and that's what Wikipedia
says too.

~~~
rectangletangle
Dynamic typing vs. static typing is on a different axis than strong vs. weak
typing. Python is a strong dynamically typed language, with some "static lite"
features introduced in Python 3.

Dynamic typing means that types can be changed arbitrarily at runtime,
compared to statically typed languages which define all types at compile time.

Strong/weak means that type coercions rarely/never happen automatically. For
instance JS has some interesting behavior enabled by weak typing `[] + [] ->
""`. Whereas Python rarely coerces things for you. The division operator in
Python 2 was strongly typed, while they changed it to weak typing in Python 3
(inline with the practicality vs. purity convention).

