Hacker News new | comments | show | ask | jobs | submit login

I encountered exactly the same issue few years ago in UIDAI in one of our large scale biometric matchers and the resolution was exactly the same. After a week of debugging I found that the libstdc++ allocator was the culprit. I found [1] and confirmed the same, which helped in fixing this issue.

The thing that was more interesting (or sad) was to know that the GCC developers didn't expect the multithreaded applications to be long running.

"Operating systems will reclaim allocated memory at program termination anyway. "

[1] https://gcc.gnu.org/onlinedocs/libstdc++/manual/mt_allocator...




Notes about deallocation. This allocator does not explicitly release memory. Because of this, memory debugging programs like valgrind or purify may notice leaks: sorry about this inconvenience. Operating systems will reclaim allocated memory at program termination anyway.

Wow. This is worth a Linus Torvalds-level rant. Whoever accepted this code into the source tree needs to be put on GNU's version of a performance improvement plan.


This is actually a fine thing to do, i don't remember where i read it, but a good analogy for trying to free memory at program exit is like trying to clean the floors and walls of a building right before it is demolished.

This is also why Valgrind separates reachable and unreachable memory and only considers unreachable memory as leaks.


How do you know the program is even supposed to terminate? Maybe it's a server, as in this case. Maybe the code will be reused someday as a subfunction or library within a larger application, instead of being launched and terminated directly by the OS. Maybe it's an embedded application in a 24/7 factory somewhere. Maybe it's on its way to the Kuiper Belt. Or maybe it's just supposed to stay up and running for longer than the average Windows 10 update period.

In any case, hiding this sort of behavior in a way that sucks down days of debugging time on the part of one expert programmer after another, after another, after another, is terrible engineering.


Freeing memory at program exit may be unnecessary, but being unable to free memory while the program is running is terrible.

On iOS and android you are expected to free whatever unused memory you can when you are notified of a low memory situation.


It's OK to do, but it makes memory analysis difficult. If your app exits with a lot of allocated memory, it's hard to tell what's a real leak and what's not.


This behaviour is one of those cache-memory-leaks. Where even though memory is reachable it is still effectively leaked because it's soaked up by some data structure and not released to the rest of the system.

So it's not a traditional leak but because memory usage would continue to grow it causes the long lived process to choke itself and die.


A good example is something like doing a "cp -a". To preserve hardlinks you'll need a mapping, trying to free that at exit can and will take time. This was an actual bug in the coreutils.


I have to wonder why the C standard library didn't include a Pascal-style mark/release allocator. A naive implementation wouldn't be much faster than free()'ing a bunch of allocations manually, but the general idea offers possibilities for optimization that otherwise aren't available to a conventional heap allocator.


Because (m)alloc predates mmap by a decade or so [1], and you expressly don't want stack-like allocation semantics when using malloc (otherwise you'd just put it on a/the stack).

[1] And you cannot have more than one heap without mmap. Without mmap, you only have sbrk = the one heap. On UNIX and those that pretend to be, anyway.


Raymond Chen often uses this analogy on the blog Old New Thing.


Years ago I read in the Perl documentation ( http://perldoc.perl.org/perlfaq3.html#How-can-I-free-an-arra... ):

> On most operating systems, memory allocated to a program can never be returned to the system. ... Some operating systems (notably, systems that use mmap(2) for allocating large chunks of memory) can reclaim memory that is no longer used ...

Because you can't return unneeded memory to most operating systems (or because it used to be that you couldn't return unneeded memory, even if that has changed recently), it isn't a surprise that by default GCC's free() and operator delete -- which are meant to be cross platform -- don't try to return that memory. Instead it's all free list management.

I do think it's silly for operators new/delete to have a separate free list from malloc()/free().


>I do think it's silly for operators new/delete to have a separate free list from malloc()/free().

They don't. This is a custom, simple, non-default pool allocator for standard containers (i.e nothing to do with new)


Thanks for the correction. But it's just as silly to create an allocator for the standard containers, when the allocator consists of little more than free list management, given that malloc/free and new/delete already do that.

It's especially silly for a library (programs may want custom memory management and libraries really shouldn't go out of their way to make that harder), and the fact that it's the standard library doesn't make it less silly.


GCC's std::allocator also doesn't do that (not for at least a decade, IIRC). It's a non-default allocator, which nobody is forced to use. It's entirely optional. The default std::allocator just uses new/delete.


Isn't this just a Free List allocator?

https://en.wikipedia.org/wiki/Free_list


Make them work on gnome?


Looks like this is just one of possible "non-default" extension allocators, that can be selected during libstdc++ compile time: https://gcc.gnu.org/onlinedocs/libstdc++/manual/memory.html#...


Facts and up-to-date documentation, how novel.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: