
Shrink Go binaries 7x with this one weird trick - FiloSottile
https://blog.filippo.io/shrink-your-go-binaries-with-this-one-weird-trick/
======
shirro
30MB binaries can be a bit of a problem for some people.

I started developing a system from home and when deploying to a VPS for
testing and production. Transferring Go binaries was like waiting for a floppy
disk to copy due to poor Internet infrastructure. Compression didn't help
much.

It turned out to be significantly faster to push to a remote git and have a
hook do the build. I think before you look to compress a go binary consider if
there aren't other options.

~~~
dietrichepp
I feel your pain. My home internet connection is 24 Mbit/s down, but only 1
Mbit/s up. I asked my ISP for more, but it's basically impossible to get.

Thank goodness for multipart uploads, at least.

~~~
shirro
I am in the same situation though my actual speed is more like 0.7 to 0.8Mbps
up. Uploading a 30MB Go binary would take nearly 6 minutes.

I guess I should be thankful I mainly work with text and not images, video or
large proprietary file formats otherwise I think I would be out of the game.

My phone is now about 5 times faster than my home Internet for both uploads
and downloads but also significantly more variable and expensive. The fibre
internet upgrade that was scheduled for last year had the technology
downgraded and the rollout delayed due to politics.

~~~
gcb0
shop for dsl. if you can still find out one. they don't advertise much these
days, buy there are still some around.

~~~
voltagex_
Shirro's probably in Australia, with terrible copper infrastructure. The
fastest _upstream_ ADSL connection is 1 to 2 megabits and SHDSL is rare.

~~~
gcb0
Cooper being rare makes it more likely that they upgrade the end points to get
3mbps or more. I'd guess that hold true for down under as it does for the
other places i know.

------
userbinator
_Note: I don 't actually believe a 30MB static binary is a problem in this day
and age_

Depends what that 30MB does, or how much of it is actually needed - it's less
of a problem if all those bytes are used in some form or another, than if a
significant portion of them are "dead code" that is never used and only serves
to take up space in memory, on disk, and in communications channels whenever
it's transferred somewhere.

 _and I would not trade (build time | complexity | performance | debug-
ability) for it_

On the contrary, smaller binaries are usually going to be better at all of
those, except perhaps "debug-ability" if you keep the debugging symbols in the
same file.

~~~
dozzie
>> Note: I don't actually believe a 30MB static binary is a problem in this
day and age

> Depends what that 30MB does, or how much of it is actually needed [....]

Moreover, let's consider a CLI tool suite. Dozen little tools, each doing
perfectly what it was designed to (think of coreutils, util-linux, sysstat,
binutils, or iproute2). Such a suite suddenly weighs 300MB, more than X.Org or
PostgreSQL (or the two combined). Does it really do more than any of these two
to justify the size? I doubt it.

~~~
userbinator
The availability of huge disks has probably skewed the perception of sizes to
some extent. 300MB is more than sufficient for an entire operating system,
_including_ a GUI and several dozen CLI tools. A full install of Windows 98
was 175MB.

On the other hand, internet speeds have not increased quite as dramatically,
so we can still relate with stories of "this app is _how_ big? And it only
does _what_?"

~~~
skykooler
Indeed. Just the other day I was trying (with no luck) to find a simple
drawing app for Android (something like Paint on windows) that was less than
18 MB. Seriously, that sort of app should not need huge executables like that.

~~~
voltagex_
I'd bet that they're shipping all kinds of graphics libraries, and probably
some support for older Android versions. What drawing app was 18MB?

~~~
hvm
Or 15MB out of those 18MB are the 100 ad libraries bundled in that serve
weird/dubious ads at best.

------
codedokode
Note that there are downsides with this method:

\- the startup time increases

\- if you run several instances of an app, they consume more memory because
code uses anon mapping instead of file mapping

\- the whole executable file is loaded to memory at startup

~~~
modeless
A bigger downside: various virus scanners flag UPX-packed executables as
suspicious.

~~~
fileoffset
Not entirely true. Most AV will only flag 'non-standard' UPX packed
executables as suspicious. Standard UPX packed binaries are easily unpacked by
the scanner and the contents scanned without notifying user.

~~~
pilif
It's anecdotal of course, but a small application of ours went from about one
false-positive per month to no false positive in over a year when I stopped
UPX packing it.

~~~
SyneRyder
More anecdata here, but I've experienced the same thing as well (not quite as
frequent as one false-positive a month though). UPX just seems to occasionally
trip up the antivirus heuristic scans until you can get your application
whitelisted.

------
lossolo
Do NOT strip Go binary:

[https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=717172](https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=717172)

~~~
userbinator
_Fixed in version golang /2:1.1.1-4_

That bug is 3 years old and seems to be architecture-specific.

~~~
frio
"Closes: 717172 Changes: golang (2:1.1.1-4) unstable; urgency=low . * Disable
stripping, it breaks go binaries on some architectures"

The "fix" is that the Debian team stopped stripping the binaries ;).

------
chetanahuja
_Note: I don 't actually believe a 30MB static binary is a problem in this day
and age_

If you expand your context from "day and age" to "country, bandwidth provider
and pricing" you might start believing the problem a bit more

------
santaclaus
> I don't actually believe a 30MB static binary is a problem in this day and
> age

Well, binary size is pretty important for embedded work. Or mobile work.

------
jonathonf
This "one weird trick" of stripping the binary, then compressing with goupx.

~~~
prymitive
Hard disk drive manufacturers hate him

------
elcapitan
Waiting for the rebuttal "ENLARGE YOUR BINARY WITH THIS SIMPLE PILL".

~~~
sov
"With this simple .PLL!"

~~~
elcapitan
Had to look that up :D

------
thejay
Op discovered exe packer.

~~~
rhexs
He's a full stack hacker, from the browser all the way down to ruby.

~~~
falcolas
Anybody who truly understands the technology and minutia behind modern
browsers strikes me as almost more interesting and impressive than
understanding Linux in this day and age.

This isn't belittling operations folks, this is more of a comment on how
complicated, complex, and obfuscated modern browsers are.

------
sofaofthedamned
Doesn't upx mess with same page merging though?

~~~
FiloSottile
As long as you don't mind having the entire uncompressed binary in memory, I
don't think I see how. Can you elaborate?

~~~
jsnell
Every instance of the program will have its own private copy of the
uncompressed data. For normal uncompressed binaries the pages of the on-disk
binary would be shared instead. (Except for any pages that get modified, which
would trigger copy-on-write. A classic example would be load-time dynamic
linker relocations).

~~~
noselasd
Sure - but that's a different thing than same-page merging.

------
jedisct1
Problem is that if you distribute UPX packed binaries, dumb antivirus will all
raise an alarm.

------
floatboth
Also useful for Haskell: [https://www.fpcomplete.com/blog/2015/05/haskell-web-
server-i...](https://www.fpcomplete.com/blog/2015/05/haskell-web-server-
in-5mb)

------
donatj
Haha, the author linked to my poorly written poorly conceived old grumbling
post. I'm honored.

I should update the post as the binaries come out especially larger in the new
versions of Go written in Go. I believe Hello Worldis up to at least 3mb.

~~~
kevincox
Isn't the output supposed to be identical between the C and Go versions?

------
pklausler
Not actually specific to golang.

------
fibo
Bravo Filippo!

------
themartorana
Honest question - mathematically, how can something be 7x smaller if x is not
pre-defined? I always find it weird that anything can be more that 100% less
than anything else because, you know, zero.

Am I way, way off base? Should I perhaps retake high school?

Edit: the smaller number is assumed 1x. Got it. Still feels weird.

~~~
michaelcampbell
You're right, but as another poster noted "2x smaller" generally means 1/2 the
original size. That usage, to me, is a horrible misuse of English, and is a
pet peeve.

You also see it with slower; "the unoptimized code is 2x slower". Huh? Do you
mean "takes 2x the time"?

I cringe every time.

~~~
maho
It makes perfect sense to me. I interpret "x" as shorthand/lazyhand notation
of the Angloamerican multiplication symbol "×", which I verbally expand to the
word "times". Furthermore, calling something "two times smaller" is equivalent
to calling it "half the size", since "smallness" is simply the logical and
mathematical inverse of "bigness", or "size". The same argument holds for
unoptimized code being two times slower, or half as fast. (Slowness is the
inverse of speed.)

~~~
michaelcampbell
> calling something "two times smaller" is equivalent to calling it "half the
> size"

This is the (IMO circular) assertion that is made by fiat. If you believe
this, then you believe it.

I _get_ the meaning, it just isn't logical to me. To use the 'x' or '*' as
multiplication should mean that many repetitions of something, not the
reciprocal of that number.

Reasonable people disagree.

------
bobwaycott
This is the second time in about a week that a post on HN has included a
variation of "this weird trick" (the other being jlongster's article dealing
with React) in the title. I can't help but parse these articles as equivalent
to all the horrible ad spam all over the web about "this one weird trick" that
will flatten your belly, tone your butt, help you make millions, give you a
bigger penis, eradicate your cellulite, and a host of other "improvements" to
your life.

Let's maybe spend a couple extra minutes coming up with less link-baity
titles.

~~~
foota
I think the title is a parody of those titles.

~~~
Godel_unicode
The fact that it's a parody doesn't preclude it being annoying. Also, it's not
actually that good a parody in my opinion. It's a click-bait title which links
to an article with a really obvious message (lose weight:eat less::shrink
binary:packer)

