
Reducing Python's startup time - vanni
https://lwn.net/Articles/730915/
======
raymondh
To be clear. I never said that start-up time wasn't important. Instead, I
pointed out that in the previously studied example, the measured impact on
start-up time of a single named tuple was a fraction of a millisecond and less
than 0.1% of start-up time.

At Guido's behest, I agreed to treat this miniscule component of start-up time
as being important and will review and implement some optimizations in the
sprints next week. I've agreed to do this even if it comes at the expense of
code clarity, complexity, or maintainability. I expect that when I'm done,
_source will be a lazily generated attribute and that much of the code
currently in the template will be pre-computed. This seems reasonable to me.

FWIW, I've devoted 16 years of my life to optimizing Python and care a great
deal about start-up time. I'm not happy about all the Raymond bashing going on
in this thread.

~~~
vanni
Thank you Raymond for all your great work.

Regarding this post and related comments, I would like to say that, as an
external observer who does not know the details of CPython development
decision-making process, phrases such as "Okay, then Nick and I are overruled"
and "At Guido's behest" sound not so good to my ears.

Maybe you and Guido are best friends, and you are half-joking with that tone,
or maybe you are responsible for that part of code, so it is your duty to
change it according to BDFL last words even if you do not agree with them... I
do not know. Anyway I think it would be better, in the open source world, if a
code change is developed by someone who thinks it is the right thing to do, or
I could not see so much difference from commercial companies and proprietary
software.

Hey, just my two cents. Keep rocking! :)

~~~
treebog
I liked those comments, and I thought they reflected well on Raymond. It's a
sign of a healthy technical organization that contributors are willing to
disagree and still heed decisions made by those higher in the decision-making
chain.

------
NelsonMinar
namedtuple is particularly weird. When you create a new type of namedtuple,
Python creates the source code for a new class from a string template and
passes it to exec() to build the class. It's a clever and relatively
straightforward bit of metaprogramming, but it's not a suprise it's not fast.
[https://hg.python.org/cpython/file/b14308524cff/Lib/collecti...](https://hg.python.org/cpython/file/b14308524cff/Lib/collections/__init__.py#l236)

(BTW if you need a quick and dirty way to make Python start up faster, see if
you can use the -S flag for your application. It skips the site-specific
initialization and has a significant impact on startup times.)

~~~
makecheck
I'm curious, given the presence of a ".pyc", shouldn't these kinds of costs be
paid primarily on first invocation "ever"?

~~~
chronial
No, because the string given to eval will not be stored in the pyc – it has to
be compiled on every import.

~~~
tankenmate
It can't be cached? Is not caching a dependency issue?

~~~
pwdisswordfish
It can't, because lol dynamic dispatch. Exercise: what does this code do?

    
    
        import collections
        collections.namedtuple = lambda _0, _1, _2=False, _3=False: "Where is your god now?"
        import b
    

b.py:

    
    
        import collections
        print(collections.namedtuple('Point', ['x', 'y']))
    

And this is not even particularly 'evil' code, in the sense of relying on
internal implementation details; that's just your bread-and-butter Python
language semantics. It prevents (or at least complicates) many other potential
optimisations as well.

------
sevensor
I'd been wondering why _source was in there. I'm a little surprised that
Hettinger thinks it's so important to retain it. It's a neat demonstration of
how to make a new data type, and I like that it's so straightforward, but I
don't buy the argument that slowing down the interpreter is important for
didactic reasons.

Edit: I'm talking about namedtuple, the implementation of which is slow
because namedtuple() has to construct _source for every new namedtuple type.
Which was clear from the article but not from my comment.

~~~
raymondh
One purpose of _source is to make the namedtuple self-documenting -- that has
been a considerable benefit over the years and has resulted in many fewer
questions and issues than we usually get about our other tools.

Another purpose of _source is so that you can write it out to a module. This
lets you pre-compute the named tuple for speed and it makes it possible to
easily Cythonize it or to customize it (in particular, people like to take out
the error checking).

Since the namedtuple() factory function was already creating the source string
as part of its operation, just storing it and exposing the attribute had
nearly zero additional cost.

That said, don't worry about it, next week I plan to make this a lazily
generated attribute so you all can sleep soundly at night :-)

~~~
nerdponx
Does this mean you're changing how Namedtuples are implemented, or just not
saving _source?

------
takeda
Seems like this article is inflammatory and totally unnecessary.

It is written in a way to pin a blame on someone (Raymond H) and also looking
at python-dev (and even on the github issue) the decision was made month
earlier (July) to go with the optimization[1], and Raymond complied[2].

[1] [https://mail.python.org/pipermail/python-
dev/2017-July/14860...](https://mail.python.org/pipermail/python-
dev/2017-July/148606.html) [2] [https://mail.python.org/pipermail/python-
dev/2017-July/14861...](https://mail.python.org/pipermail/python-
dev/2017-July/148611.html)

------
eesmith
I have grown to seriously dislike 'namedtuple'. It seems great, and it's so
easy to use, but there's so few places it can be used without dragging in
future backwards-compatibility baggage.

Let's say you have a light-weight object, like a "full name" which can be
represented with a "forename" and an "surname". (Yes, there are many names
which don't fit this category. This is a toy example.) So you write:

    
    
      from collections import namedtuple
      FullName = namedtuple("FullName", "forename surname")
      name = FullName("Edward", "Smith")
    

So simple! So clean! No "self.forename = forename" or need to make your own
__repr__.

Except you also inherit the tuple behavior, because this isn't really light-
weight object but a ... let's call it a heavy-weight tuple.

    
    
      >>> name[0]
      'Edward'
      >>> name[1]
      'Smith'
    

This mean people may (okay, are likely) do things like:

    
    
      forename, surname = name
    

or even

    
    
      for forename, surname in names:
        ...
    

Even if that's not what you want in your API. If you care about backwards
compatibility, then if you replace the namedtuple with its own class then
you'll have to re-implement __getitem__, _replace, __len__, and count -
methods which make no sense for a simple name class.

Or, if you decided to change the API, to make it be:

    
    
      FullName = namedtuple("FullName", "forename middlename surname suffix")
    

then you'll again break code which used the full indexing API that you
originally implicitly promised, simply by saying this is a namedtuple.

'namedtuple' has one good use - migration from a tuple object to a real object
while allowing backwards compatability to the tuple API. For example, the old
tuples once returned from os.stat() and time.gmtime().

It's also fine if it will only be used by code where you have full control
over its use and can either prevent inappropriate API use like destructuring
assignment, or can update all uses if you want to change the API. For example,
a light-weight object used in a test-suite or internal helper function.

Otherwise, don't use it. Spend the extra few minutes so in the future you
don't have to worry about backwards compatibility issues.

~~~
sametmax
Only if one is misusing namedtuple: it should be used for iterable things,
like csv lines, db rows, coordinates, etc. but not for a person. That makes no
sense.

If what you need is a cheap, quick container, then you want
types.SimpleNamespace.

~~~
eesmith
"but not for a person. That makes no sense."

I wrote 'toy example' for a reason. While it does not make domain sense, it's
easy to understand and it highlights the problems I wanted to show.

"it should be used for iterable things"

Really? This article concerned the use of namedtuple in

    
    
      functools.py
      21:from collections import namedtuple
      403:_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
    

The cache_info() method returns a _CacheInfo instance to the user. Where/what
are the "iterable things"?

If a future version adds another attribute, it will break most code which uses
destructuring assignment.

(For that matter, I can't see why it make sense to use a namedtuple for this
case. Why not just:

    
    
      class _CacheInfo:
        def __init__(self, hits, misses, maxsize, currsize):
          self.hits = hits
          ...
    

? Who cares about immutability here?)

Or for another case, here's a Python function which returns a class derived
from a namedtuple:

    
    
      >>> from urllib import parse
      >>> result = parse.urlparse("http://www.python.org/")
      >>> for term in result:
      ...   print(result)
      ...
      >>> for i, term in enumerate(result):
      ...   print(i, repr(term))
      ...
      0 'http'
      1 'www.python.org'
      2 '/'
      3 ''
      4 ''
      5 ''
    

Again, where is the "iterable thing"? I mean, the ParseResult instance is
iterable, but my point is that it makes no sense to iterate over those
fragments.

If the Python standard library gets it wrong, what is the right solution?

"If what you need is a cheap, quick container, then you want
types.SimpleNamespace. "

You mean the one where the documentation at
[https://docs.python.org/3/library/types.html#types.SimpleNam...](https://docs.python.org/3/library/types.html#types.SimpleNamespace)
says "SimpleNamespace may be useful as a replacement for class NS: pass.
However, for a structured record type use namedtuple() instead." ?

I do not think you are correct.

~~~
sametmax
There are plenty of issues with the stdlib. Part of it don't respect the PEP8,
there are inconsistencies with argument types and orders, some of it is even
unusable for real life usages (ex: csv in python 2.7) and we keep it for
legacy reasons.

The fact is, the use cases you are showing are wrong, and the one I'm talking
about are the proper ones. It's ok to make mistakes, we all do a lot of them.
And we will have to pay for those.

If you are interested in the subject, last month we had a huge debate on the
Python mailing list about a way to have a proper official immutable but non
interable structured record to replace named tuple and avoid those abuses. And
also to maybe include something like attrs in the stdlib.

~~~
eesmith
It would be nice if you could show me a proper example, as what you wrote
doesn't match with my understanding of how it is used nor with how it should
be used.

My argument is to resist the temptation to use namedtuple for anything except
a migration from tuples to named attributes. While there is some overlap with
what you wrote, they are not the same. I would like to know what I'm missing.

~~~
sametmax
A proper use for namedtuple:

    
    
        @property
        def coordinates():
            x = self.calculateX()
            y = self.calculateY()
            return x, y
    

this could become:

    
    
        Point = namedtuple('Point', ('x', 'y'))
    
        @property
        def coordinates():
             x = self.calculateX()
             y = self.calculateY()
             return Point(x, y)
    

It stays compatible with the previous implementation. as you can still do:

    
    
        x, y = stuff.coordinates
    

But now can do:

    
    
        point = stuff.coordinates
        point.x
    

The same example apply for a row a from timeseries in a database or a line of
stats in a csv.

But now imagine you have a response from a ldap server returning a list of
authorized person. You could manipulate those persons in a dictionary. Some
people may want to have a nicer way to access them, with the look up syntax.
You can be tempted to use a namedtuple here, but that make no sense, as a
person is not iterable. The alternative is to use empty class, but it's very
verbose.

But now you can use types.SimpleNameSpace:

    
    
        >>> class Person(types.SimpleNamespace): pass
        >>> p = Person(name='foo', age=12) # no __init__ to write
        >>> p # free repr
        Person(age=12, name='foo')
        >>> p.age # nice access syntax
        12
    

The good thing is that if you ever need to extend Person to later have a more
complex behavior, you can just do so as it's a regular class.

The bad things are:

\- you may need something immutable

\- you may want some checks done to restrict the attributes used

Which is why currently people abuse namedtuple. We don't have a good story for
those. Hence the debate on the mailing list.

Nevertheless, it's good to remember that the Python community has a philosophy
of "we are all consenting adults here" and a good track record for following
it. immutability and attributes restrictions are not definitive show stoppers
in Python, and SimpleNameSpace is a decent solution while waiting a purer to
come up.

~~~
eesmith
Your example with Point() is exactly what I said was a good use case, for
apparently the same reasons why I said it was a good use case.

I said "'namedtuple' has one good use - migration from a tuple object to a
real object while allowing backwards compatability to the tuple API. ... It's
also fine if it will only be used by code where you have full control over its
use and can either prevent inappropriate API use ... Otherwise, don't use it."

You replied "Only if one is misusing namedtuple".

I interpreted this as an objection to my use guidelines. From what I can tell,
you are in complete agreement with me.

------
makecheck
Moving "import" statements into dynamically-controlled blocks goes a long way
in my experience, despite being flagged by tools like "pylint".

Buried imports free the interpreter from doing something until it is actually
required; really nice if you just want to run "\--help" and not wait 4 seconds
for utterly unrelated modules to be found. It also creates this interesting
situation where a script can technically be broken (e.g. module not found) but
you don't _care_ as long as the _part_ you're using is OK.

Grouped imports are undoubtedly nice for purity and easily seeing dependencies
but they may not be smart in a dynamic language. It is still pretty easy to
"grep" to find imports if you're trying to track dependencies.

~~~
nerdponx
Would be nice if CPython had an option to enable "lazy imports".

------
Boxxed
One side benefit to removing the use of "eval" is that namedtuples will then
be pickle-able. I'm surprised this hasn't come up: every now and then I hit
some weird data I want to quick-and-dirty serialize but can't because there's
a namedtuple buried deep inside it.

~~~
xapata
Namedtuples are pickle-able when created in the global scope, just as any
dynamically created class. The main section of the collections module
demonstrates that feature (read the source, Luke).

Removing eval from tuple would not solve your issue. You'd need to instead
remove eval from pickle, of sorts. The trouble is that pickle tries to look up
your namedtuple by name and can't find it in your module.

------
mikepurvis
One of the comments below the article mentions this, but in my experience on
Linux and Mac, by far the biggest culprit is pkg_resources and its nasty habit
of spidering the filesystem:

[https://github.com/pypa/setuptools/issues/926](https://github.com/pypa/setuptools/issues/926)

There are hacks to get around it, but it's a deep hole.

~~~
orf
People often complain that using `entry_points` in setup.py is slow. I did
some profiling and it came down to it importing some `pip` submodule
(pip.vendor.something), which imported basically the whole of pip.

:(

------
jordigh
Mercurial did a lot of work to reduce startup time via its demandimport
module.

[https://www.mercurial-
scm.org/repo/hg/file/tip/hgdemandimpor...](https://www.mercurial-
scm.org/repo/hg/file/tip/hgdemandimport/demandimportpy2.py#l1)

Basically, it's a lazy loading of all imports. You can write `import foo` but
it won't actually be imported until you do `foo.whatever()`.

It's a crutch, and it's true that hg still pays the overall startup cost of
Python. However, even with "45 times slower than git", the situation is not so
dire:

    
    
      jordi@chloe:~$ time hg --version
      Mercurial Distributed SCM (version 4.2.2)
      (see https://mercurial-scm.org for more information)
    
      Copyright (C) 2005-2017 Matt Mackall and others
      This is free software; see the source for copying conditions. There is NO
      warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
      real	0m0.096s
      user	0m0.084s
      sys	0m0.008s
    
      jordi@chloe:~$ time git --version
      git version 2.1.4
    
      real	0m0.001s
      user	0m0.000s
      sys	0m0.000s
    

I really can barely perceive a difference between 0.096s and 0.001s. Since it
really is a one-time startup cost, it's not like we can even say that this
difference accumulates and that hg is overall 45 times slower than git.

Pierre-Yves also has an interesting talk about all of the tricks that have to
go into hg in order to make it fast with Python. There's stuff like doing
`method = object.method; method()` instead of doing `object.method()` over and
over again to avoid paying the cost of method lookup and so forth:

[https://2015.pycon.ca/en/schedule/53/](https://2015.pycon.ca/en/schedule/53/)

~~~
coldtea
> _I really can barely perceive a difference between 0.096s and 0.001s. Since
> it really is a one-time startup cost, it 's not like we can even say that
> this difference accumulates and that hg is overall 45 times slower than
> git._

It's bad for many things -- like showing VCS status on your shell prompt or
IDE status line, where you repeatedly start hg.

100ms+ delay every time you hit enter in your shell is noticeable (and that's
just the version, now doing an actual "dirty" check takes more).

~~~
Too
Mercurial has something called command server[1], it's essentially a daemon
listening on sockets to receive commands, designed just to avoid the cost of
startup. I had to use it once in a CI system where we did invoke hg thousands
of times for some reason i can't even remember.

That can also be used for frequent polling like shell status.

[1] [https://www.mercurial-scm.org/wiki/CommandServer](https://www.mercurial-
scm.org/wiki/CommandServer)

~~~
ngoldbaum
In fact I have hg locally aliased to chg and have not noticed any issues with
it. Just much faster command completions and an actually usable shell that
includes hg annotations.

------
wyldfire
BDFL says:

> Concluding, I think we should move on from the original implementation and
> optimize the heck out of namedtuple.

I remember reading this on the mailing list and thinking "yes, namedtuple
deserves optimization because it's an excellent resource".

For my use cases, I don't think I notice Python's startup time. When I write
Python code it's usually not performance critical. When I do write
performance-critical Python code, I usually care about total runtime and PyPy
is usually a win here. Aside: if you use PyPy, you should probably be using
namedtuples. IIRC it models those much better than it does dicts. And IMO
namedtuples model many common data structures better than dicts do.

Just because I don't feel the pain of Python's startup time doesn't mean we
shouldn't try to optimize it. I think I find my way into some esoteric
Python/CPython corners but FWIW I've never needed namedtuple's "_source" attr.

~~~
mikepurvis
Ruby's idea of a hash where the keys are colon symbols seems to really strike
a sweet spot here— it's fast and storage-efficient like a tuple, but has the
flexibility of a dict.

~~~
sevensor
One big advantage of using namedtuples, aside from the immutability I
mentioned elsewhere, is that you can rely on a namedtuple having particular
fields. So if you're using dictionaries, you have to worry about KeyError
wherever the dict is used, but if you're using namedtuples, you can't create
them without assigning values to all of the fields. Dictionaries are still
great for their intended purpose -- to represent a mapping -- but namedtuples
are nicer if what you really want is a lightweight object.

~~~
mikepurvis
Setting default values for the namedtuple requires subclassing it, though,
which is its own little bag of fun. See all the wailing and gnashing of teeth
on SO:
[https://stackoverflow.com/search?q=namedtuple+subclass](https://stackoverflow.com/search?q=namedtuple+subclass)

~~~
JelleZijlstra
If you're using Python 3.6, it's a lot easier now. You can do:

from typing import NamedTuple

class Point(NamedTuple): x: float y: float = 0.0

and then Point(1.0).y == 0.0.

~~~
sevensor
That's quite nice!

------
Redoubts
> The approach of generating source code and exec()ing it, is a cool
> demonstration of Python's expressive power

Man, I don't see many positive opinions of exec out there.

> I think we should move on from the original implementation and optimize the
> heck out of namedtuple. The original has served us well. The world is
> constantly changing. Python should adapt to the (happy) fact that it's being
> used for systems larger than any of us could imagine 15 years ago.

I really wish the BDFL could say the same for CPython itself. This continued
insistence of treating python as a teaching language with a teaching
implementation is really weird.

------
jxcl
Raymond Hettinger, if I remember correctly, is the person who originally wrote
NamedTuple, and he's not shy about reminding people during his talks at
conferences. I wonder if his pride or other personal feelings about that
particular module could be clouding his judgment over whether or not to
optimize it?

~~~
mjburgess
I think it's more to do with him primarily teaching python (iirc) rather than
working on large systems with it.

It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to
teach than Py2 and thef. bad. Crazy.

EDIT: OK, if not primarily teaching, spending a lot of his time teaching it.
That's a big part of his motivation + worth knowing that when assessing his
arguments.

~~~
WoodenChair
> It's the same strain of rant Zed Shaw had of Py3, ie., that its harder to
> teach than Py2 and thef. bad. Crazy.

Why is wanting something to be easy to teach crazy?

~~~
mjburgess
Programming languages are tools for developers. They are taught in the service
of software development. Optimizing for teaching has it backwards.

It can be a factor: but to place it higher than startup time is prioritising
the one-off teaching experience of a few over the repeated development
experience of many. For a tool _for_ development. Crazy, no?

~~~
hueving
Ability to teach is going to be directly correlated with ability to maintain
as a business. It's harder to hire people when the bar for learning the skill
is higher.

The vast majority of software development is not hitting any kind of scaling
or performance bottlenecks. So optimizing for simplicity isn't as crazy as you
think.

~~~
mjburgess
I think python's start up time is going to correlate much more strongly with
business success than whether Raymond's 12th slide in his 16th deck requires
five minutes more time.

~~~
yen223
Absolutely not. Python always had a suboptimal performance story relative to
other languages, but people use it anyway.

Businesses use Python because it is easy to pick up, not because it's fast

------
jorams
> The approach of generating source code and exec()ing it, is a cool
> demonstration of Python's expressive power

This sentence is very surprising to me. If the best way to implement something
is to literally build a _string of source code_ and pass it to the
interpreter, to me it means the user is unable to really express what they
want in the language.

As a (terrible) example, imagine a language whose number addition operator
only works on literals.

    
    
        1 + 2
        => 3
    
        a = 1
        b = 2
        a + b
        => ERROR!
    

It does, however, have a string concatenation operator that also works on
variables and a way to read user-provided numbers into a variable, as well as
some kind of eval function. What would you do to add two user-provided
numbers? Something like the following:

    
    
        foo = read number!
        bar = read number!
        baz = concatenate x, ' + ', y
        quux = eval baz
    

The same thing could be stated about this language...

> The approach of generating source code and eval-ing it, is a cool
> demonstration of <this language>'s expressive power.

...but it clearly wouldn't make sense. Functions like exec and eval are an
escape hatch for when there is no sensible way of expressing something in the
language.

~~~
comex
I’d say that the existence of an escape hatch for things that can’t be
expressed natively is itself a form of expressitivity. It’s one that’s near
ubiquitous among dynamically typed languages, so it doesn’t say much that
Python has it, but it still provides a favorable comparison to some static
languages.

That said, this particular use case for eval is basically macros lite: the
code being evaluated doesn’t generally depend on input to the resulting
program, so it could have been expressed as a transformation to the module
source code, i.e. a macro. And a true macro would be able to provide more
natural syntax, whereas namedtuple requires stuffing the field names into a
string (or using a wrapper based on some metaclass thing, but the syntax for
that isn’t great either). Thus, languages that have macro support, both
dynamically typed (Lisp) and statically (e.g. Rust), could be called more
expressive in this respect.

------
epberry
Ah, slow startups on unittests is a big headache. This is especially apparent
on big projects like Django where you might as well leave your desk and grab
coffee when the unittests start, even for small projects. That said, I can't
help but think that this is just an inevitable tradeoff of Python. You want
great expressive power? You have to sacrifice speed. Awesome constructs like
namedtuple make this worth it imo.

~~~
mfukar
I can't help but think if a one-time initialisation cost is bogging a process
down, there are bigger problems than that initialisation cost.

~~~
coldtea
There are processes that you might need to run frequently. I gave an example
above: showing the output of a Python script on your shell prompt (e.g.
mercurial status).

~~~
majewsky
I moved from Python to Go for rendering my shell prompt for much the same
reason: [https://blog.bethselamin.de/posts/latency-
matters.html](https://blog.bethselamin.de/posts/latency-matters.html)

~~~
coldtea
Funnily I did exactly the same -- rewrote parts of a powerline (shell prompt
status enhancements) implementation that I used and was written in Python, and
turned it into Go.

But it still has to call into Python to get my mercurial status (we use
mercurial at work so I need to have that).

------
chubot
I agree with optimizing Python's startup time, and I agree that namedtuple is
weird and should be changed.

But I doubt that the namedtuple change will noticeably decrease startup time
for most applications (after having experience with this problem for 10+
years).

From a comment over 3 years ago:

 _[Python interpreter startup does] random access I /O (the slowest thing your
computer can do) proportional to (large constant factor) x (num imports in
program) x (length of PYTHONPATH)._

[https://news.ycombinator.com/item?id=7842873](https://news.ycombinator.com/item?id=7842873)

I don't think that instantiating a Python parser 100 times for exec() is going
to compare to that. I guess the difference is that I'm thinking about the cold
cache case -- namedtuple might be noticeable in the warm cache case.

And there are many many command line tools that start slow because they're
written in Python, not just Mercurial.

Mercurial is actually one of the best because they care about it
(demandimport) and they don't have too many dependencies. IIRC some of the
Google Cloud tools take 500-1000+ ms to start because they are crawling tons
of dependencies with long PYTHONPATH.

~~~
eesmith
If you are really interested in improving the startup cost then the first
thing is to put everything, including the Python standard library, into .zip
files, so it can be zip imported. That removes a lot of the directory/stat
overhead. The comment you pointed to did not do that optimization.

For example, I supported a 10 year old set of CGI scripts running on a machine
with Lustre filesystem where every filesystem metadata lookup was painfully
slow. (See ehiggs' comment at lwn.) I spent time to trim away every import to
the bare minimum.

Once you do that, other factors become more significant. (Remove your biggest
problem and something else becomes your biggest problem.)

If the view is that a few milliseconds here and a few milliseconds there isn't
a problem, well, those milliseconds add up.

'import scipy' (which depends on numpy) takes 3x longer to import than Python
itself does. I have replaced dependencies on scipy when the overhead of
importing the one function I needed took longer than my program took to run.

There's an old story along these lines, recounted at
[https://www.folklore.org/StoryView.py?story=Saving_Lives.txt](https://www.folklore.org/StoryView.py?story=Saving_Lives.txt)
:

> One of the things that bothered Steve Jobs the most was the time that it
> took to boot when the Mac was first powered on. ... One afternoon, Steve
> came up with an original way to motivate us to make it faster. ...

> "You know, I've been thinking about it. How many people are going to be
> using the Macintosh? A million? No, more than that. In a few years, I bet
> five million people will be booting up their Macintoshes at least once a
> day."

> "Well, let's say you can shave 10 seconds off of the boot time. Multiply
> that by five million users and thats 50 million seconds, every single day.
> Over a year, that's probably dozens of lifetimes. So if you make it boot ten
> seconds faster, you've saved a dozen lives. That's really worth it, don't
> you think?"

How many people use Python? How many times is it started per day?

~~~
batbomb
The other way around this is containers or caching filesystems.

------
Walkman
At my company we changed the interaction between two daemons by calling a
command line script instead. It caused a 300ms delay in every single AJAX call
and broke a dozens of end-to-end tests.

I just tried with Python 2.7 and 3.6 and the hello world startup time is 100ms
which is awful lot.

------
evmar
I suspect one factor for why Go is so popular within Google is that (due to
complex path hacks as mentioned here as well as a network file system) Python
startup is so slow.

On an app I worked on the unit tests took >10 seconds to boot on each run.
Among the languages available to Google developers, Go ends up being the
lightest weight, despite there being plenty of other lighter languages
available outside. (This is not the place to start an argument about language
plurality.)

------
hathawsh
I feel like the namedtuple issue could be solved cleanly by adding a feature
I've wanted for a long time in Python: some simple way to cache high-level
data and code in generated .pyc files. Today, there's no way to do that;
Python compiles the code but evaluates nothing until execution time. I'd like
my .pyc files to contain some precomputed expressions and code, to reduce
startup time. I probably ought to discuss this on python-ideas.

------
fractalb
If the namedtuple is used so heavily, why not make it a part of built-in
functions?

------
ericfrederich
It's funny. Go on #python on freenode and ask a question about named tuple.
Consensus there is that they should be avoided in favor of real classes or
attrs decorated classes.

------
first_amendment
It's unfortunate that the new version still creates __new__ using exec().
Doesn't seem necessary at all. Instead of generating the method as a string
with the argument names filled in, why not use use a combination of * n and *
* kw?

------
est
IMHO this namedtuple is non-issue compared to buildout.

buildout tends to append millions of path to os.path

Annnnd your python project start time requires 1 minute or more because each
import requires scan millions of directories.

------
camus2
My question is, is it worse or better with Ruby,NodeJS, Perl,Java or Go? I
remember NodeJS scripts being extremely slow to start in general, way slower
than Python.

~~~
majewsky
Sorted from slowest to fastest: (all single measurements on my Arch Linux
notebook)

    
    
      $ time java &> /dev/null
      java &> /dev/null  0,09s user 0,03s system 120% cpu 0,102 total
      $ time ruby -e 42
      ruby -e 42  0,04s user 0,01s system 99% cpu 0,053 total
      $ time python -c 42
      python -c 42  0,03s user 0,00s system 98% cpu 0,030 total
      $ time perl -e 42  
      perl -e 42  0,00s user 0,00s system 92% cpu 0,005 total
    

I don't have NodeJS at hand to test with. Note that Java produces a help
message (which I sent to /dev/null), but that (hopefully) shouldn't affect the
timing that much.

Go produces compiled binaries and, as such, exhibits no such startup delay.
Startup times for Go programs that I tested are in the same ballpark as Perl
above, i.e. not reasonably measureable with a millisecond-precision tool like
time(1).

------
wiradikusuma
heh, when I read it, I thought it was about reducing _Java_ startup time to be
on par with Python's.

------
NewEntryHN

        $ time python -c ''
        real	0m0.071s
        user	0m0.064s
        sys	0m0.000s
    
        $ time python -c 'import functools'
        real	0m0.073s
        user	0m0.052s
        sys	0m0.016s
    

I mean...

~~~
d33
It's not exactly a reliable benchmark for many reasons...

------
sillysaurus3

      $ time python -c 'print(1)'
      1
    
      real	0m0.067s
      user	0m0.015s
      sys	0m0.025s
    

Is it slow? That was Python 2.7.10, and for me Python 3.6 is similarly fast.

~~~
benhoyt
On my system (macOS) I see a minimum of 0m0.034 for Python 3.6.1 and a minimum
of 0m0.022 for Python 2.7.13 -- so Python 3.6 is 50% slower.

This is a somewhat contrived example, however. As the article notes, it's
worse when you have a lot of imports (and I don't know how that'd differ
between versions 2 and 3). I remember talking to some folks at Google who work
on the Google Cloud CLI, and they said that Python startup time is a constant
problem for their (large) codebase. Especially when it's used for tab
completion, because it might run every time you press tab, and the user wants
instant results.

~~~
mikepurvis
We're dealing with this on a CLI build tools project I hack on (catkin_tools).
It uses plugin discovery based on entry_points, and short of caching plugins
in a homedir file, it's basically impossible to make it fast enough to use
with something like argcomplete.

Relevant ticket:
[https://github.com/catkin/catkin_tools/issues/297](https://github.com/catkin/catkin_tools/issues/297)

