
When the memory allocator works against you - bostik
https://glandium.org/blog/?p=3698
======
rini17
How about trying pypy? It has different GC/memory allocator and the
2.7-compatible branch is pretty much drop-in replacement (Unless you need
something like PyQt).

------
bogomipz
The author states:

>“Import manifest” phase (which, in fact, allocates and frees increasingly
large buffers for each manifest, as manifests grow in size in the mozilla-
central history)."

I am not familiar with the reference of a "manifest" or importing them at
least not in the context of memory allocation. Can someone shed some light on
this?

Also right after the author states?

>"To put the theory at work, I patched the python interpreter again, making it
use malloc() instead of mmap() for its arenas."

If I understand correctly the author is trying to remove the possibility of
entanglement but doesn't the malloc() glibc library routine translate into an
mmap() system call? I am not sure how that would help disambiguate memory
allocators then.

~~~
eternalban
> "Import manifest" phase ... in the context of memory allocation [+].

+of git operations.

~~~
bogomipz
Sorry, I don't understand your reply. Could you elaborate?

~~~
eternalban
A manifest is an element of a Git repository. Importing a manifest is a git
operation. So OP is looking at memory patterns per distinct git ops.

~~~
bogomipz
I see, that makes perfect sense. I'm not sure how I missed that, thanks.

------
quickben
>"I hadn’t optimized for memory use in the first place."

It is git, it is made to grow and get into billions of string comparisons.
Anyways...

C/c++ would have probably been a better language for his problem.

Simply prealocate as much as needed as a working buffer and eliminate the need
to profile libc and patch your interpreter altogether.

~~~
Kenji
I concur. If the memory allocator causes problems and you use a language with
managed memory/a garbage collector, you're doing something wrong.

~~~
Filligree
An aggressive, copying garbage collector would have eliminated this problem
from the start. People often argue that GCs have unavoidable overhead, but
it's bounded overhead; they defragment your program with every GC cycle.

~~~
Kenji
GC does have unavoidable overhead, in particular, it makes the execution times
of your binary extremely unstable and high-variance. Not to mention, it has no
idea how your program uses memory, such that it is bound to use a generic
approach while with manual management, you can get real and high performance.

~~~
pg314
> it makes the execution times of your binary extremely unstable and high-
> variance.

Not necessarily, it depends on your allocation patterns. The same can be said
for using malloc.

> Not to mention, it has no idea how your program uses memory, such that it is
> bound to use a generic approach

Ditto for malloc.

> with manual management, you can get real and high performance.

Using a GC does not mean you can't manually manage allocations. E.g. say you
are allocating and deallocating a lot of Point objects. You keep a free list
of Point objects that you can recycle. If you need a Point, you first see if
there are any in the free list. If so, you pop the first one off the list and
reuse it. Otherwise you allocate a new one. When you're done with a Point, you
push it on the free list.

~~~
jstimpfle
>> Not to mention, it has no idea how your program uses memory, such that it
is bound to use a generic approach

> Ditto for malloc.

malloc != manual management

> Using a GC does not mean you can't manually manage allocations. E.g. say you
> are allocating and deallocating a lot of Point objects...

But that's not using a GC. It's manual memory management.

~~~
buzzybee
That's aggressively missing the point. If you design the program architecture
around manual memory techniques, the runtimes are stable even in a GC. If you
use malloc willy-nilly throughout core algorithms, your runtimes become
unpredictable.

~~~
jstimpfle
But my point was simply that you can't defend GC by contrasting to manual
management, if you do manual management.

> If you use malloc willy-nilly throughout core algorithms, your runtimes
> become unpredictable.

First, where did I say "use malloc willy-nilly"? Second, do you have
experience with that? I haven't much, but I haven't heard your claim before.

Of course "willy-nilly" should not be done with real-time constraints. But I
have written a substantial algorithmic program (no RT constraints) in that
style when I didn't know better, and it was very well performing. (probably
glibc's malloc was optimized for my allocation patterns).

The main problem I see with that style is that each malloc has memory
overhead, and that it leads to unmaintainable code.

~~~
Kenji
Actually, to address realtime constraints... In my job, I work on systems with
realtime constraints and even in less critical parts (latency of ~100
microseconds is still acceptable) we had to eliminate all malloc calls. Mostly
because malloc on our platform essentially acquires one big lock for memory
operations and if some other process has that lock, you get nasty delays or
even priority inversions. Allocate a big chunk beforehand, preferrably in the
form of ring buffers, and then operate on these if you need precise timing and
performance.

~~~
jstimpfle
... or use a version of malloc on that preallocated thread-local chunk. At
least if the only problem is locking overhead.

------
draw_down
Good thing you didn't optimize prematurely ;)

------
__s
ITT: people being snide about scripting languages over a post digging into a
glibc malloc mishap

~~~
Kenji
Did he really find a bug in glibc malloc? He didn't write that. This sounds
like a bug in the python runtime and possibly in the offending program itself.

~~~
malisper
I don't fully understand the post, but I don't think the author found a bug
anywhere. The main discovery the author made was that CPython uses its own
allocator on top of memory allocated by mmap. This leads to a difference
between the reported allocated memory by malloc and the memory used by the
program. The author still hasn't discovered a root cause for the large amounts
of memory allocated.

~~~
glandium
Author here. The last graph actually shows that the problem still happens with
CPython's allocator disabled.

Also, to clarify, the amount of memory allocated is actually _decreasing_ (in-
use bytes), while the memory allocator requests more and more memory from the
system (system bytes), presumably because of fragmentation.

~~~
Kenji
>The last graph actually shows that the problem still happens with CPython's
allocator disabled.

Excuse my ignorance, but couldn't it still be a problem of the python runtime?
I.e. they use malloc wrong, etc?

------
crististm
People never learn. News at eleven.

Just the other day was reading N. Wirth objecting on using a resource lavishly
just because it was cheap. And just now, in the same front page of HN, some
guy wants programmers to stop calling themselves engineers because they
aren't.

Well, let's build a bridge using all the iron we can get our hands on (iron is
very cheap you know); or even better, let's use all the RAM in the computer
with this program of ours.

~~~
mikeash
I'm pretty sure "real" engineers would be very much in favor of using a cheap
resource lavishly, if the tradeoff is worth it. Adding a million dollars to
the design cost in order to save $100,000 in resource costs isn't a good idea,
and a good engineer will design accordingly.

The difference is that engineers have codified this and call it "safety
factor." Your house is built to carry perhaps ten times the load it truly
needs to, because it's cheaper to throw in extra materials than it is to build
it to hold precisely what it needs. An airplane is built with much thinner
margins, because saving materials is far more valuable, and it's well worth
the extra time and attention to.

~~~
crististm
Look up Hamming's course on "Learning to learn". An optimal design is not
robust. You add a safety factor and this addition is wasteful in some sense
(thus not optimal) and necessary in another. Do you design a building so that
the toilets are optimally used? Likewise for planes.

But just because you have enough of a cheap resource (RAM) it is not a given
that using it in excess is either optimal or robust.

~~~
TeMPOraL
Also, most users aren't really aware what's using up the resources - only that
their computer gets slower with time. I actually wish Windows would start
naming & shaming applications - e.g. by showing a balloon when the system
slows down, that would say that application X and Y are consuming most of the
resources.

As it is now, CPU, RAM and storage usage are perfect externalities for
developers. The result of their abuse is user frustration, and reduction of
the number of things people can do on their computers. That, and wasting user
time. Unfortunately, there's no way that would make the developers suffer
costs for wasting user's resources, so the only thing that lets us have lean
software is engineering competence and human decency (both of which seem to be
in short supply).

