Hacker News new | past | comments | ask | show | jobs | submit login

Missing technique: rewrite the performance-critical parts of your Go programs in a different language, and use Cgo to make them accessible to Go code via the standard C ABI. K.I.S.S.

This isn't simpler at all. Managing CGo is many times more difficult than a little optimizing. If you're able to write the hot path more performantly in C or similar, then you're perfectly capable of optimizing the Go version--the principles are the same and you don't need to incur the costs of crossing the CGo boundary nor do you need to reason about the memory/ownership handoff between languages or any of the other half dozen downsides of CGo.

This is not simple in any way. cgo is convenient for being able to use "lib*" packages, but calling these packages is not free, and certainly writing cgo for performance reasons is not likely to be a gain.

Go is plenty fast for most things, notably things that work well with a GC... but cgo doesn't make the GC go away.

This is a good way to ensure a relatively slow (by comparison) security review for your PRs as every dumb string manipulation and every allocation needs to be checked.

'I am deliberately ignoring techniques that require significant effort, or large changes to program structure.'

Hopefully that's /s, as debugging CGo stuff is a royal pita. ;)

also the overhead of calling out to c can actually be quite high: https://www.cockroachlabs.com/blog/the-cost-and-complexity-o...

While some overhead and increased debugging effort may be inevitable, it's a mistake to blame them on the use of Cgo itself; the root cause is Go's custom ABI and user-level-threading ("goroutines") model. And I really have to dispute OP's claim that the techniques they mention do not "require significant effort, or large changes to program structure" of their own. In many ways, rewriting some portion of the code can actually be simpler.

Well, it may be true that the "blame" for why CGo is slow is up for debate, it's just a fact that it's rather slow to use.

Go isn't unique in this; there are many languages with runtimes that require similarly intensive amounts of copying and context conversion before C code can run on whatever the data is. However, most of those languages, like CPython, are themselves slow enough that the penalty isn't as noticeable against the general background noise. (In general CPython requires a lot more copying too; Go is closer to C struct and array semantics and can more often get by with some form of memcpy, the internals of Python look nothing like that.) Go is fast enough that it's much easier to get into scenarios where in a tight loop you're spending 90% on CGo overhead if you're not careful with data flow. For those languages that have to copy a lot out of their runtime and are also fairly fast, they'll face the exact same issues. It's not really a "Go" issue per se, but the challenges faced by any language that wants a runtime significantly different from C.

(One of the miracles of Rust is building an environment and runtime that isn't stuck on C's limitations but at the same time can still speak to C really, really cheaply. Plenty of languages have one or the other of those, but there aren't very many that have both. I'm not sure there's any other language that has threaded that particular needle so cleanly.)

> Go isn't unique in this; there are many languages with runtimes that require similarly intensive amounts of copying and context conversion before C code can run on whatever the data is.

I agree, and some of this overhead is even inherent in legitimate, foundational choices such as VM-interpreted/"managed" code (as in Java/.NET; but Python does this as well) or the use of tracing GC (which requires some strict discipline on heap contents, so as to enable the GC itself to reliably "trace" and discover the semantics it cares about). So, I'm definitely not saying that the choices made as part of Go's design are consistently wrong here!

Indeed, Haskell is in a very similar place overall; the "fibers" that GHC uses in its compiled code are implemented via async code underneath, much like Go's goroutines, and Haskell's performance is also well within an order of magnitude of pure C-like code. So the issues you describe are quite well-understood in that context, and they are definitely not regarded as a "reason" to avoid the use of C FFI when performance requirements call for it. But describing Cgo itself as something that's high-overhead and should not be used for that reason is misleading to an even stronger extent, and that's what I was objecting to in the grandparent comment!

Well said. Also worth noting that most of the languages like CPython are very slow because many optimizations are now prohibitively difficult given the "easy C-interop" constraint.

Isn't that more because the choice to support C interop by giving C access to interpreter internals, which is the easiest-implemented option but not the only way to get easy-to-use C interop.

That could be. Although I'm not familiar with any languages that are fast and have a GC and easy C interop. At least none of the major VM languages or (non-embedded) scripting languages. Maybe D or some other similarly non-mainstream language.

Lisp? Common lisp's ffi faculties are good, from all I've heard.

Could be. I’m not that familiar.

This is Go not Python.

Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact