
Diagnosing Memory “Leaks” in Python - todd8
http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html
======
hassy
Nice read, even though I haven't written any Python in a long time. Memory
management in modern OSes is very complicated and there isn't a simple answer
to "how much memory is my process using exactly". [1] Memory growth / usage
isn't usually something to worry about unless memory growth is constant under
load and/or OOM killer kicks in.

Also consider that `heapy` will probably only report on objects created by
your Python code, and not any memory taken up by native code in the
interpreter itself or any shared libraries.

1\. For example see:

\- [http://stackoverflow.com/questions/860878/tracking-
actively-...](http://stackoverflow.com/questions/860878/tracking-actively-
used-memory-in-linux-programs/872456)

\- [https://mail.gnome.org/archives/gnome-
list/1999-September/ms...](https://mail.gnome.org/archives/gnome-
list/1999-September/msg00036.html)

\- [http://bmaurer.blogspot.co.uk/2006/03/memory-usage-with-
smap...](http://bmaurer.blogspot.co.uk/2006/03/memory-usage-with-smaps.html)

------
IgorPartola
Once I had to fix a severe memory leak in piece of Python code. None of the
available tools revealed anything useful. I ended up just adding debug
statements to the hot loop to see which step caused the memory usage to jump.
Should have known the answer: a bad C extension was leaking memory. Lesson
learned.

~~~
danudey
libzmq apparently has a bug where it creates structures to handle connections
every time you connect, and it cleans them up after processing data. This
means that if you connect then disconnect without sending data, you leak
memory.

It's trivial to run a "for i in …" and cause a zmq app to leak gigabytes of
memory in a matter of seconds. Highly problematic.

~~~
toong
Are you referring to
[https://news.ycombinator.com/item?id=10175751](https://news.ycombinator.com/item?id=10175751)
?

------
ThePhysicist
I encountered the "memory hogging" behavior of Python processes once, where I
was sure that my GC worked correctly and that I released all unused objects
but still the memory of the process would keep growing. I also remember having
this problem with a C++ once as well. It doesn't seem a problem though since
the OS should normally release the memory if it's needed by another process.
Still, this kind of behavior can definitely drive you nuts.

BTW Celery is usually not a good fit for long running processes, because if
you have many of those processes running in parallel within a production
system it will get really difficult restarting the Celerey daemon as it will
have to wait for all these processes to stop (during which no new tasks can be
processed). Why restart at all you ask? Well, restarting is necessary to
reload the code, as it is not recommended to use the autoreloader in a
production system. This problem persists even when using the multiprocessing
module btw (as the author suggested), since on Linux Python uses fork() to
create a new process, thereby just copying the whole memory of the given
process.

~~~
ledil
What do you consider as alternative to celery?

~~~
ThePhysicist
If you find one, let me know ;) I'm looking for something myself currently.

There's RQ ([http://python-rq.org/](http://python-rq.org/)) but it seems to
have a similar design as Celery (just a simpler architecture) so it probably
suffers from the same problem.

A good solution would be to have a series of workers that can launch new
independent Python processes for each task, e.g. using the subprocess module.

~~~
aidos
I have long running jobs (say, 5 minutes on average - up to an hour). I
originally used celery (after picloud shut down) but it just doesn't work well
with those charcteristics. Each worker reserved an extra job so it was
impossible to get good cpu utilisation.

I switched to rq and it's all been much easier. The behaviour is easy to
understand and it's easy to inspect redis to see what's going on.

In terms of the code restart angle - I'm fairly sure you can effectively
restart the workers. They run as a single process that forks to do work. Each
copy you run only has a single worker, so you need to run multiple instances
yourself. If you kill the parent it waits until the child has finished the job
it is on before terminating.

I could be wrong about some of the details. I'd recommend giving it a shot. I
must have run 100,000s of jobs through it now and I haven't had a single
issue.

[http://python-rq.org/docs/workers/](http://python-rq.org/docs/workers/)

~~~
ThePhysicist
Thanks, that sounds interesting!

------
cheez
Kind of surprised the poster didn't know that operating systems often hold on
to memory.

> we noticed that the memory of the celery process was continuing to grow.

Doesn't look like there was any bad outcome related to this observation. Was
any process not getting the memory it wanted?

~~~
ldarby
There are in fact bad outcomes of continued memory growth - firstly there is a
catastrophic performance degradation when processes start getting swapped to
disk, then when swap space runs out Linux's OOM killer starts killing
processes in an attempt to free up memory. Kind of surprised you didn't know
this.

~~~
cheez
Except that didn't happen, according to the post.

~~~
ldarby
The post says: "Indeed, the processes grew from their initial size of 100MB,
slowly, all the way up to 1GB before we killed them."

The important part is that they manually killed the processes before they let
it start swapping.

~~~
cheez
Again, they didn't actually have anything bad happen. Bad: processes couldn't
get memory. Not bad: I was using 3 GB of memory.

------
mkesper
Problem seems to be solved in Python 3.3+:
[http://bugs.python.org/issue11849](http://bugs.python.org/issue11849) The
article mentions some interesting disgnostic tools, though.

~~~
Drdrdrq
The linked issue probably has nothing to do with the case (see last 3 comments
on issue page).

Nice summary of python memory debugging tools though. I would add dowser to
the list too.

------
hydrogen18
If you think you actually have a memory leak in python, the first thing to do
is to recompile without the internal memory manager in python. This avoids the
problem the author is addressing.

~~~
jsmeaton
Can you expand on this a little more or provide some links that explain? What
happens to memory if you remove the memory manager?

~~~
hydrogen18
It just uses the provided malloc & free implementations from the underlying C
standard library.

------
vvanders
Kinda surprised the author didn't know about compacting GCs that solve memory
fragmentation(at the cost if having a GC).

------
rasz_pl
tldr: uses python 2.x, discovers known, fixed long ago behaviour

------
curiousjorge
import resource import objgraph

these are some golden modules that I never knew about

