
Programs compiled by Go 1.11 allocate an unreasonable amount of virtual memory - networkimprov
https://github.com/golang/go/issues/28114
======
ahh
Unreasonable according to you. :)

The nice thing about virtual memory is that it's, well, virtual. It costs you
almost nothing until you've touched it. (Fun exercise for the reader: measure
the kernel overhead for an unused 1 TiB VMA.) But creating huge spaces--that
terabyte mmap wasn't theoretical--that stay untouched is hugely
algorithmically useful, especially for things like malloc implementations.

Why does it bother people? Two reasons. First is mlock to avoid swap. This is
solvable in much better ways--I'm a fan of disabling swap in many cases
anyway. Second is that, absent cgroups, it's difficult to put hard limits on
memory usage in Linux. So people, looking under the streetlight, put limits on
virtual usage, even though that's not what they care about limiting! Then they
get angry when you break it. My refrain here, as in many cases (see for
example measuring process CPU time spent in kernel mode): "X is impossible"
doesn't justify Y unless Y correctly solves the problem X does.

(I spent years in charge of a major memory allocator so this is a battle I've
fought too many times.)

~~~
exabrial
You cannot possibly be serious:

package main

func main() { for { } }

Using 100mb+ of memory?

~~~
outside1234
It is not "using" but allocating the addresses. The committed memory will be
vastly smaller. This is how virtual memory works.

------
kccqzy
Check out GHC 8.0+ where the same program will allocate a terabyte of virtual
memory ;)

That said, I don't find it unreasonable at all. Just reserving some bits in
the address space isn't unreasonable. It makes the real allocation code
simpler.

~~~
xfalcox
Yes, we are running a Haskell based HTTPS redirector and people noticed this.

------
weberc2
Impact:

> The significant increase in virtual memory usage is usually not an issue,
> however security sensitive programs often lock their memory, causing a far
> greater performance degradation on low-spec computer hosts.

~~~
exacube
I'd love some further explanation here:

what does locking memory mean in this context? For what purpose do security
sensitive programs lock their memory? what is the performance degradation that
happens with low-spec computers?

~~~
deathanatos
> _what does locking memory mean in this context?_

I think the person is referring to the mlock() and mlockall() functions (or
equivalents on other OS), which keep pages resident / prevents pages from
being paged out. Forces them to remain in RAM.

It can be used to, e.g., prevent an encryption key or password from being
swapped out to disk, where it might then be recoverable. (Personally, this is
why I encrypt swap.)

> _what is the performance degradation that happens with low-spec computers?_

Locking a larger portion of RAM means less room for the OS to page out unused
pages and free up the space for other programs.

While one can try to selectively lock buffers with sensitive data with
mlock(), you have to be sure they aren't copied into other buffers that aren't
locked (and could thus be subsequently paged out). If you're writing a UI
program that displays or receives those in a widget, this might be harder (you
might not have access to the internal buffer of the widget, as it is an
"implementation detail" of your library), and locking the entire process might
be a simpler solution (albeit being a bigger hammer).

~~~
monocasa
My understanding was that mlock doesn't really keep the page from being paged
out, since it can't even begin to do that in the hibernation case, or if
you're running as a VM who's guest RAM wasn't mlocked.

What it does is keep a canonical version of the page in memory. That's useful
being able to deterministically touch a piece of memory, but it doesn't really
help you as far as making sure the page never touches disk.

~~~
deathanatos
I hadn't considered hibernation, and indeed, a deeper reading of the manual
confirms that hibernation doesn't count, which is rather interesting (given
the implications of hitting disk), but I don't really see a good way around
it, short of aborting the hibernation, or providing a mechanism to inform the
program that those pages were lost. The man page (later, annoyingly, after its
initial description) notes this:

> _Memory locking has two main applications: real-time algorithms and high-
> security data processing. Real-time applications require deterministic
> timing, and, like scheduling, paging is one major cause of unexpected
> program execution delays. Real-time applications will usually also switch to
> a real-time scheduler with sched_setscheduler(2). Cryptographic security
> software often handles critical bytes like passwords or secret keys as data
> structures. As a result of paging, these secrets could be transferred onto a
> persistent swap store medium, where they might be accessible to the enemy
> long after the security software has erased the secrets in RAM and
> terminated. (But be aware that the suspend mode on laptops and some desktop
> computers will save a copy of the system 's RAM to disk, regardless of
> memory locks.)_

I would be rather disappointed if a hypervisor swapped out my guest (at least,
in a context like AWS; I suppose if you're just running qemu on your laptop,
that's a different matter), but I hadn't considered that either, and it is
certainly possible.

~~~
monocasa
IMO, the right answer is to better define your threat model. Are you concerned
about someone pulling the HDD and reading the swap? Use an FDE scheme that
covers your swap too. Are you concerned about someone getting access to the
swap file programmatically? At that point they have so many other ways of
slurping the memory out of your process that it's a lost cause.

------
CJefferson
I do this in some programs, it makes many pieces of coding much easier if I
know I will never have to move some objects as they grow.

The ideal for me would be a function that marked some memory as "these
addresses are taken, do not give them out to malloc, or anything else", but
which still required me to actually "ask" for the memory before using it, so
it didn't look like I was using 64gb of memory at startup. Is that possible?

~~~
boomlinde
_> The ideal for me would be a function that marked some memory as "these
addresses are taken, do not give them out to malloc, or anything else", but
which still required me to actually "ask" for the memory before using it, so
it didn't look like I was using 64gb of memory at startup. Is that possible?_

So you want to reserve memory, basically the way malloc does, depriving any
other process of that memory, but you then want to reserve it _again_ , and
also somehow hide that the memory has been reserved?

What exactly is the benefit except making it not look like you're using the
memory? Actually, what's the benefit of that in itself?

~~~
CJefferson
No, I want, in my address space, to say to the kernel "when you decide what
bit of my memory map to return from mmap and friends, don't use these
addresses.". That shouldn't cost anything, except storing the blocks I want
reserving.

This is surprisingly hard to do in user space. You can tell mmap where to put
allocations, but not where not to put them. Also it's hard to control what
your Libc's malloc will do.

~~~
klodolph
You can do this with mmap(). Just use PROT_NONE, you'll get a mapping but you
can't read, write, or execute it.

Sure, it will show up in the "virtual memory" usage of your process. That's
just how the virtual memory accounting works.

