

Go Execution Modes - Rooki
https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ

======
personZ
The dogma about preventing the sharing of pointers with C is a bit concerning.
I've made use of that mechanism, to fantastic effect, albeit understanding the
risks and consequences. The notion that it must be prevented because bad
things can happen if you aren't careful isn't a starter, as there are
essentially zero people doing that who've had problems. Instead a lot of
people have enjoyed fantastic productivity because of the similarities of the
implementations, and the synergy that allows.

But the team wants move to a compacting GC. Fine, add pin/unpin idioms. This
ground has been well covered by other languages. Don't completely destroy a
very productive mechanism because in some oddball cases it might not work.

~~~
crawshaw
You can do your own pinning, which is what I do in gobind[1]. The basic idea
is to keep a map in Go of the pointers:

    
    
        var ptrs = make(map[uintptr]unsafe.Pointer)
    

When passing a pointer from Go to C:

    
    
        var p unsafe.Pointer = ...
        id := uintptr(p)
        ptrs[id] = p
        C.Fn(id)
    

When C returns the pointer to Go to be used, run it through the ptrs map again

    
    
        //export GoFn
        func GoFn(id uintptr) {
            p := ptrs[id]
            // ... use p
        }
    

Don't forget a cleanup function where you delete(ptrs, id). If you want to
pass the same pointer multiple times to C and keep it comparable, you'll need
a second map.

All of this is requires being very careful, but the hard part is the notion of
holding references to memory outside the realm of the GC. I don't believe a
runtime-assisted pinning API can do any better than what you can do yourself
with a map.

[1]
[https://godoc.org/code.google.com/p/go.mobile/cmd/gobind](https://godoc.org/code.google.com/p/go.mobile/cmd/gobind)

~~~
niemeyer
That's a fine model when you want to hand a reference to a Go value, just to
be used in Go itself, but it's not actual pinning. The data can still move,
which means you cannot use the data in C, for example to share a buffer
without copying into another temporary buffer managed by the C allocator.

~~~
crawshaw
That's right, you cannot dereference the pointer from C. If you want to use
allocated memory from both C and Go, you'll need to allocate it in C.

~~~
ominous_prime
What's the use-case for passing a pointer to C that can't be de-referenced?

~~~
niemeyer
You can pass it back into Go for dereferencing, at which point you can use the
uintptr => unsafe.Pointer map, and it will be correct since the moving GC will
have preserved the unsafe.Pointer address at the right location.

The pattern is useful. It just doesn't handle the most common case, which is
passing a buffer or a simple output parameter into a C function.

------
pauldix
This would be awesome to have. +1!

With InfluxDB we want to give users the ability to define custom functions and
it looks like this could be used to let people do that in Go or any language
really. Whereas right now we're limited to doing something like using LuaJIT.

------
axaxs
I would love to see the gc compiler support building dynamically linked
executables against a libgo runtime Ala gccgo. Unless I'm misreading, this
case doesn't seem covered. I love the standalone binary deployment ease as of
now, but on a machine with tens of go applications, the binary size redundancy
gets old. Or did I miss something?

~~~
FooBarWidget
We're in an age where everybody is wasting gigabytes of memory through
virtualization, and nobody cares. Would those few megabytes of memory saved by
dynamically linking against libgo really be worth it? For past 5 years or so,
it seems people value ease of deployment much more than efficiency.

~~~
ColinCera
You have a point, but on mobile and embedded devices, nobody is "wasting
gigabytes of memory through virtualization" and not caring.

~~~
laumars
It doesn't cost gigabytes of RAM, but Android applications do run on a Java
virtual machine.

But I'm being massively pedantic.

------
Rapzid
I think the more relevant news in there for most is the support for
dynamic/runtime loading. The title here makes it sound like it's just for
dynamic linking but I read the article to article to be sure and was
pleasantly surprised :)

~~~
Rooki
Good point, I updated the title to include the actual title of the doc but
left the rest to avoid confusion.

On topic: I'm really excited to see this, even though it's made clear this is
just a plan and not a promise it's still great news for things like Android
development.

------
lobster_johnson
> Building Go packages as a shared library

Yes, please! We have a few potential Go projects which are blocked by the
inability to separate Go code into plugins.

I have toyed with the idea of forking child processes and implementing plugins
via pipe I/O, but that's a very unfriendly API interface, and I don't feel
very inspired. (Go isn't that great at POSIX stuff like forking, either.)

~~~
_stephan
Why do the plugins have to be dynamically loaded? Would a source-based
plugin/package system be an alternative?

~~~
lobster_johnson
Consider a data processing app. It calls little data-processing modules
written in Go. You want to open-source it because it's general-purpose and
reusable; but you don't want to open-source all your in-house, proprietary
data-processing components, too; they're not of use to anyone but yourself, or
they may be something you can't open-source for other reasons.

That means the core project cannot list its plugins as dependencies. You will
instead want to be able to load plugins from a configuration file.

(The problem is actually equally awkward in languages that have package
systems, such as Node and Ruby. In Ruby, for example, a program can only see
gems declared in the Gemfile; same problem with npm-shrinkwrap.json.)

I suppose you could set up a special build system where plugins are expected
to be installed into a certain folder so that they're statically built in,
with some kind of hook so that they're registered into the core. That's the
Nginx approach. I'm not a big fan, especially since this method of building
things need to be reinvented for each app that needs a plugin system.

You could, of course, invert the relationship and make this system a library:
To set it up with your plugins, you write a small program that runs the
system's main entrypoint with a list of plugins. But that just makes it harder
to configure, build, use and deploy.

------
bkeroack
I realize it's just an early draft, but it's interesting (alarming?) that the
proposed plugin API has no facilities for plugin discovery or separate
namespaces. Nor any apparent ability for the plugin consumer to introspect
into plugins.

I suppose the namespace issue can be mitigated by a strict naming convention
(something like "[application_name].[plugin_name]"), but the others are kind
of a bummer.

I don't have a whole lot of positive things to say about the Python setuptools
API (perhaps better called "lack-of-API internal interface"), but at least
it's possible to dynamically load python plugins from arbitrary locations in
the filesystem and programmatically discover their contents.

~~~
ianlancetaylor
The plugin API uses file names, so discovery would be at a higher level. Go
packages impose their own namespace, so there shouldn't be any namespace
issues. Introspection is useful, but would be a matter of examining the files
using existing packages like debug/elf; it would not be part of the plugin
API.

Basically the plugin API is isomorphic to dlopen/dlclose. Everything else is
built on top of that.

~~~
bkeroack
OK, if true that makes sense. I was interpreting the Name parameter to Open()
as being something like a global plugin name rather than a filesystem
location.

That combined with the Plugin struct (which hopefully would have something
like an iterable Map of exported symbols) resolves the issues I was concerned
about. Nice.

------
easytiger
Is there a mirror? For those of us in the corporate wasteland this is not
accesible

------
JuiceSSH
Interesting that it will allow compilation of PIE executables. This is a
mandatory requirement for any binaries run on newer versions of Android (L+).

~~~
dmm
Is it for security reasons? I know that OpenBSD uses PIEs so that the runtime
can randomized address space locations.

------
frik
The doc doesn't mention a timeline or Go version when we may see the new
planned features.

~~~
enneff
That's unknown. Work will likely begin in December after Go 1.4 is released,
so the earliest we'll see this work is in Go 1.5.

