Hacker News new | comments | show | ask | jobs | submit login

This is the relevant bug: http://code.google.com/p/go/issues/detail?id=909

As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.

Can anyone knowledgeable about compilers/GCs say why GO went with a conservative garbage collector? From any GO source code its trivial to pick out pointers from values. Is this information hard to preserve at runtime?

Probably ease of implementation. You can just grab the Boehm conservative GC, slap it in, and bang- you've got GC. You can even add GC this way to C/C++ programs.

Unfortunately, Boehm has draw backs- because it's getting no help from the compiler, it can't tell integers from pointers. So it has to treat everything that looks like it might be a pointer as a pointer, even if it's an integer (or floating point number). Which means that it's possible for garbage to not be collected, because there is an integer that happens to have the same value as the address of the garbage object. And, of course, once you can't collect that object, you can't collect all the objects it refers to (including false pointers), and so on.

The odds of this happening are a function of what percentage of the virtual address space is in use- once some critical threshold is reached, the amount of garbage that can't be collected due to false pointers just explodes. On 32-bit platforms, I've seen this happen with heap sizes of only a few hundred megabytes. And the advice to work around this is exactly what the responder said- use less memory, don't use large ints (which are more likely to be mistaken for pointers), etc. Also, the problem goes away (for the time being) on 64 bits, because the percentage of memory used drops. A terabyte of memory on a 64-bit system is the same fraction of the total address space as a kilobyte of memory is on a 32-bit system.

A conservative GC is easier to implement. No other reason.

I recall reading something along the lines of it being hard to use a precise GC due to the the "unsafe" package, but I am not knowledgeable at all.

Here's an early LtU thread where someone predicted it'd probably have to use a conservative GC due to some of the addressing features: http://lambda-the-ultimate.org/node/3676#comment-52560

Isn't this issue also going to be a problem with Linux upcoming X32 ABI?

Yes, because x32 ABI's pointers are still 32 bits, so pointer values are in a small enough range that they can fool the GC into thinking they are legitimate int values.

I know early JVM garabage collectors were also "conservative", but I don't recall JVMs running into these sorts of problems. Maybe folks are just using more of the 4G address space these days than back in the day of conservative GC JVMs?

Within the JVM, object references are strictly separate from integers. Object references are stored within a structure they call an OOP, short for "ordinary object pointers", and the gc works by inspecting the contents of oops and potentially changing them if objects are moved around in memory (as happens if the gc is moving things between generations, or doing heap compaction). If you're curious about it, you can get some information about OOPs on the page discussing the CompressedOops flag implementation: https://wikis.oracle.com/display/HotSpotInternals/Compressed...

I've never heard of a JVM that uses this kind of GC. As far as I know, every Sun JVM has used explicit "this is a pointer" to identify things for GC.

Ah, you're right. Its only the thread stacks that were conservatively scanned, not all of memory (see "Mostly Accurate Stack Scanning" for some indirect evidence of such). Its not clear to me if modern JVMs are still conservatively scanning stacks or not. But clearly its much less of a problem than conservatively scanning all of memory.

Publicizing the bug may speed the fix along. Here's the real test: is hitting the front page of HN going to make a difference?

I don't think this one is lacking interest, but an obvious fix. It looks like Russ Cox did commit a fix for some portion of the cases. But a better fix would either need to significantly modify the way things are laid out to reduce the odds of false positives, or else move to a precise GC.

(Precise GCs are possible in C-like languages, but trickier to implement. Here's a recent paper on one: http://www.cs.utah.edu/~regehr/papers/ismm15-rafkind.pdf)

Von Neumann architecture falsely lead us to the path with the singular notion of "memory". We need to distinguish at the hardware architecture level between "working memory" and "persistent memory".

I doodle all the time when working. Pristine set of diagrams emerges from the chaos. I wish my computer would play along with this regime ..

I haven't mentioned downvotes to date, but the down votes on this is puzzling. Would the down voters care to comment as to why the comment is deemed down vote worthy? (I would think those who work/grok on memory managers do get the point.)

I didn't downvote it, but it mostly read like rambling nonsense to me, so that might be why. Maybe write it so it's clear to people who don't work on memory managers all day?

I've worked on memory allocators, and I can't make sense of it.

I think the idea is to differentiate between mutable and immutable data, ie. "worked on" and "final". I don't know what that would give you with regards to GC though.

"Resist complaining about being downmodded. It never does any good, and it makes boring reading." http://ycombinator.com/newsguidelines.html

Brief HN discussion on that paper: http://news.ycombinator.com/item?id=586858

Applications are open for YC Summer 2018

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact