

On Distributing Command Line Applications: Why I Switched From Ruby to Go - reactor
http://codegangsta.io/blog/2013/07/21/creating-cli-applications-in-go/

======
rlpb
As a non-Rubyist, I share your pain of having to use Rubygems (and pypi, and
CPAN, and all the others). However:

"Since Go is a compiled language, binaries of your app can be precompiled for
each platform that you wish to distribute for."

For me, this is worse. There's no audit trail. No way for me to verify that
the binary of your app is the same as the source you provide. So there's no
way I'm running your binary on my system. I simply cannot trust it with my
user account.

Not only is this bad from a security perspective, it's also bad for
maintenance. Over time, many end up in a situation where others cannot rebuild
binaries from source at all, since binary distribution becomes the norm and
the source->binary mechanism isn't maintained except on the developer's own
system. Look at the Java community for an example of this. This is a problem
because it means that others cannot easy be a member of the community working
on the source.

If you want to promote a community around the source (like Github does), you
must also provide an easy way to rebuild binaries from the source.

Unfortunately, there's no good answer here.

Distributions like Debian have solved this problem - there are source
packages, and there's a standard way of building them to get the binary
packages. But there's quite a bit of arcane crud that's built up over the
years, making it difficult to learn and understand Debian packaging. And the
solution isn't cross platform.

Each language community has produced their own solution (Rubygems, Python
eggs, CPAN, etc) but none of them work with the packaging provided by any
other language. To me, this may be cross platform, but not being cross
language is just as big a problem.

~~~
pkulak
Nothing is stopping you from building the app from source instead, just like
every other open source project. But at least with Go you have the _option_ of
just downloading and running a binary if you trust the author.

~~~
rlpb
This is true. I do have that option.

But I very rarely trust the author. Most of the time I've never heard of the
author. This is a good thing, and exactly why Github has been so successful.
By nature, if there's something obscure that I need doing, and I find some
software or software module that lets me do it, then chances are I won't know
the author.

This means that downloading and running the binary is suddenly a very obscure
use case. It should be for you, too.

~~~
mdwrigh2
> This means that downloading and running the binary is suddenly a very
> obscure use case. It should be for you, too.

A rare use case for you, but definitely not an obscure one in general.
Besides, trusting the author, in this case, only matters if you're willing to
audit the source to verify that nothing malicious is occurring -- something
I'm not going to do in general. I mean, realistically, when was the last time
you audited your network drivers? your OS? your browser?

~~~
rlpb
You don't have to verify the source yourself. Others can do it. I trust the
Debian maintainer for my package. He's a real world person who has had his key
signed by other Debian developers.

The fact that the source can be verified also reduces the risk of malicious
modification.

It's not necessarily just the author who might be malicious, either. The
author's computer where he runs his compiler might be compromised.

------
riobard
Not just command line apps. Any time you need to deploy on multiple machines,
Go saves you a lot of trouble.

At work we use primarily `buildout` [1] to deploy Python apps with complex
dependencies. It is rather slow and fragile. Additionally, PyPi and various
external repositories are not very reliable, and we ended up setting up local
mirrors. Nevertheless, the process is not very pleasant.

We are experimenting with Go in some smaller projects, and so far the
experience is great! We can build on a development machine (OS X) with all the
dependencies version controlled (using git-subtree) along with our code, and
cross-compile a static binary for Linux for deployment. Deploying is simply
copying the binary to the production machines (usual configuration management
is done via Puppet).

Besides, Go can efficiently make use of multiple cores (remember to set
GOMAXPROCS correctly), and we can just deploy one instance of an app per
server, unlike one instance per core as in Python's case (I know there's
multiprocessing, but it's bad for operation and management). This greatly
reduces memory overhead if the app is big (e.g. consumes a couple hundred of
MB upfront and you have 32 cores per machine = a few GB's memory is wasted).

Only problem is that Go's 3rd-party libs are not as comprehensive as Python's
yet, and many packages we rely on do not have mature/reliable counterparts in
Go. So we are still using Python for our core stuff, but smaller projects that
do not require many external packages can be done in Go elegantly.

[1]: [http://www.buildout.org/en/latest/](http://www.buildout.org/en/latest/)

------
delinka
He leaves out discussion of a crucial part of app distribution: dynamic
libraries.

While you can't create and link with dynamic libraries in Go, you can call
into C or C++ libraries. Go binaries are self-contained as long as all your
code is Go. But if you're depending on a C or C++ library, you still have to
worry about this aspect of distribution.

~~~
rwcarlsen
It is now possible to statically compile+link C libraries into your Go binary.
The sqlite3 package ([http://godoc.org/code.google.com/p/go-
sqlite/go1/sqlite3](http://godoc.org/code.google.com/p/go-sqlite/go1/sqlite3))
is a good example of this.

~~~
archgoon
Another aspect is that some opensource licensing schemes may require you to
link dynamically. The LGPL, for example, requires you to do this unless you
release the rest of your source code.

------
codegangsta
Blog author here. Long story short I got frustrated with distributing Ruby cli
apps and decided to switch to Go for speed and distribution.

I also created a nice cli library built on top of the Go flag package. I've
been having a lot of fun in Go so far and encourage anyone who hasn't given it
a shot to try it out.

[https://github.com/codegangsta/cli](https://github.com/codegangsta/cli)

~~~
johnbellone
Made the same decision as you recently.

The only pep-peeve of mine is that I'm not a huge fan of using JSON as a
configuration mechanism. I much preferred having a Ruby file for
configuration. My middle ground here is using a system ruby via /usr/bin/env
ruby and generating the necessary JSON output for people that want the nice
syntax.

------
kinofcain
This also makes go applications trivial to deploy. What used to be a massive
amount of plumbing and bootstrapping for a ruby environment with multiple app
server instances behind a reverse proxy with all the associated configuration
and dependencies can now be little more than pushing a binary to a (nearly)
stock machine instance. It really simplifies ops.

~~~
FooBarWidget
I really don't see what having multiple app server instances behind reverse
proxy has anything to do with Ruby vs Go. App servers behind reverse proxy is
an architectural decision. You can write a Go web app that must be put behind
a reverse proxy, or you can write a Go web app that is supposed to act as its
own web server. Likewise, your Ruby app does not _have_ to run multiple app
server instances behind a reverse proxy. You can run it together with your
main web server (Apache or Nginx) and you can run a single instance instead.
See [https://www.phusionpassenger.com/](https://www.phusionpassenger.com/)

Of course, installing dependencies still matters. For Ruby, you can just use
the Brightbox Debian/Ubuntu packages.

~~~
kinofcain
All the mainstream ruby web frameworks require something like nginx in front
of multiple instances of something like thin. You can use something like
phusion, unicorn or rainbows that do forking for you instead of launching
multiple processes by hand, but you still have to configure them and they're
still running multiple app instances.

Go's built-in http library, and all the associated frameworks, run multi-
threaded out of the box, and are fast enough that you don't need to put nginx
or haproxy in front of them.

I love ruby, I've been building and deploying rails apps since 1.0, the ease
of deploying go apps is precisely the type of thing the Ruby/Rails community
cares a lot about: making developers' lives easier.

~~~
FooBarWidget
It is not true that they must run multiple instances. Rails has been
multithreading-capable for years. Other Ruby frameworks have been
multithreading-capable for much longer. With Phusion Passenger Enterprise, not
only do you not have to setup reverse proxying, the sole instance can run
multithreaded.

As for "fast enough that you don't need to put nginx or haproxy in front of
them" \- the point of reverse proxying is not performance, it's security. If
anything, reverse proxying makes things _slower_ (theoretically) because the
kernel has to make one extra copy of the data. You reverse proxy stuff so that
nginx or haproxy can properly handle I/O management and concurrency, and so
they can sanitize HTTP headers, not because it gives you a performance boost.
If Go frameworks are multithreaded by default, instead of events, then Go too
can benefit from reverse proxying.

If you don't want to think about any kind of reverse proxy setup, there's
Passenger Standalone in the Ruby world. You type 'passenger start', and you
have a fully-functional, production ready server listening on any port you,
that does not require reverse proxying.

~~~
kinofcain
"the point of reverse proxying is not performance, it's security."

Not true. You absolutely need forking/multiple processes on rails deployments
to enable even a small number of concurrent requests. If phusion wasn't
forking and load balancing between those running instances, your performance
would be far lower.

EDIT: See this article from the Passenger folks on tuning a passenger
deployment, it should be more clear after reading that how Passenger is
dealing with concurrency (by running multiple app instances and load balancing
between them): [http://blog.phusion.nl/2013/03/12/tuning-phusion-
passengers-...](http://blog.phusion.nl/2013/03/12/tuning-phusion-passengers-
concurrency-settings/)

~~~
FooBarWidget
I wrote that article. :)

The concurrency tuning explained in that article has got nothing to do with
reverse proxying. You don't _need_ multiple processes either: the article says
that you can use multithreading, and especially in the case of I/O-bound apps,
that's enough. If you want to leverage multiple CPU cores however, that's a
different story, but it depends on your Ruby implementation. If you use JRuby
or Rubinius then threads are enough. But that is still unrelated to the
question of whether to reverse proxy or not.

------
mwcampbell
I agree that the self-contained nature of compiled Go programs is one of Go's
most appealing characteristics. In fact, I think you were too diplomatic
toward Ruby, and most other popular languages. In my opinion, to insist that
distributing a program should not be easy is to expect too little of our
tools. I can identify two main deficiencies that make it hard to distribute
programs:

1\. Lack of a module system that scales well to large programs while still
using static linking. According to the talk "Go at Google: Language design in
the service of software engineering"
([http://talks.golang.org/2012/splash.article](http://talks.golang.org/2012/splash.article)),
this is one of the main problems that the Go developers aimed to solve.

2\. Too much dynamism in the language, such that a packaging tool can't tell
with certainty what a program needs, and just as important, what it doesn't
need. Go is static enough that this simply doesn't apply. Dart is more static
than JavaScript, but still allows some dynamism. The Google Closure tools
solve the problem by subsetting JavaScript. Is there any such subset for
Python or Ruby? (I don't count RPython, since AFAIK that's only intended for
PyPy itself.)

So when choosing a language for a program that one expects to distribute to
many users, I think it's a good idea to take these things into account. This
may mean choosing a language that lacks some dynamic features that many of us
find convenient. (Compile-time metaprogramming can compensate; too bad neither
Go nor Dart has it.) Go looks like a pretty good language for servers and
command-line utilities.

------
earlz
I've written about this before, but this guy definitely wrote it better. I
experience a very close relation between this and Ruby vs C#(Mono). I love
Ruby as a language, but the pain of using Rubygems and random things requiring
Ruby <1.9 and <1.8 (And setting up rvm to manage that) is an absolute pain.
Python is slightly better, but I still have a python2 and python3 installed.

This is why I love mono. Things may not be backwards compatible, but I've
never had a problem where something wasn't forwards compatible. And I think
this is a result of it being compiled. The compilation step decouples the
language and syntax from the actual execution of the code. This means that
compiled applications can have drastic and breaking syntax improvements, but
still be forward compatible on future runtimes. The same can not be said of
interpreted languages unless you somehow work a version number into the
execution, and even then it makes implementation of the languages much harder
because future versions must support every version of the syntax

------
shurcooL
Related to cli and Go, if you ever want to quickly execute Go funcs from a
terminal [1]:

    
    
        $ goe go/parser 'ParseExpr("5 + 7")'
    

[1]
[https://github.com/shurcooL/goe#examples](https://github.com/shurcooL/goe#examples)

~~~
astrodust
What it needs is a `-e` flag:

    
    
        ruby -e 'puts 5 + 7'

~~~
shurcooL
Can you please elaborate on how that would help?

Is it just for familiarity/similarity? I kinda see the 'e' from ruby -e going
after the go to form goe. goe -e would be just redundant, wouldn't it?

~~~
astrodust
Perl, Ruby, Python, and many others have a succinct "evaluate this string"
command-line flag. It's so commonplace it's to be expected.

------
avtar
_After attempting the distribution of a command line application via Rubygems
I came to the following conclusion: Any application not intended exculsively
for Rubyists should not be distributed via Rubygems._

Another option could also be to convert gems (and dependency gems) to deb or
rpm packages
[https://github.com/jordansissel/fpm/wiki/ConvertingGems](https://github.com/jordansissel/fpm/wiki/ConvertingGems)

As far as I know, the Foreman project utilizes this approach
[http://deb.theforeman.org/pool/precise/stable/f/foreman/](http://deb.theforeman.org/pool/precise/stable/f/foreman/)

~~~
codegangsta
That is definitely an option, and it is essentially what we did for our linux
support when we were building out a cross platform toolchain for mobile games.
The tricky part was supporting Windows, Linux, and Mac all at once. Again, not
suprised that distribution was hard, there were just a couple things that made
distribution in Ruby a more painful experience

------
pnathan
In a prior job, I did a decent amount of Python distributions. I had it
simple: 1-2 python libraries plugging into a single app which was distributed
on a .deb server. Even that was not particularly fun. I can _definitely_
understand the appeal of a build n run system.

As a user, I find rubygem based installs rather horrific (same for pypi &
cpan).

n.b.: I'm sure there are tools out there that will appropriately massage your
app and put together a app.zip package for you. But IME those sorts of tools
are a lot of work to get going.

------
conroy
One feature I miss from distributing on RubyGems or PyPI is easy update
functionality. Thankfully, go-update[1] looks to solve the problem by making
it easy to create self-updating Go binaries.

[1]: [https://github.com/inconshreveable/go-
update](https://github.com/inconshreveable/go-update)

------
_pmf_
How many architectures are officially supported by Go?

~~~
codegangsta
Supported architectures listed here:
[http://golang.org/doc/install#requirements](http://golang.org/doc/install#requirements)

~~~
dmm
There are two implementations of go: gc and gccgo. The link above describes
gc. gccgo is built on gcc and supports many more environments, at least in
theory. gccgo will run on addtional processors like powerpc, mips, and sparc
and other operating systems like solaris. These aren't necessarily well tested
though.

------
izietto
Why don't compile Ruby using f.e. LLVM?

~~~
TylerE
Because it's kinda pointless. The thing that makes Ruby (very) slow isn't the
runtime (there might be at most a 1.5x - 2x speedup there), it's the language
design. All that dynamism and runtime reflection is really a performance
killer.

~~~
gngeal
_it 's the language design. All that dynamism and runtime reflection is really
a performance killer._

So why are Lisp and Smalltalk, which are equally dynamic and reflective, even
faster?

~~~
greiskul
Ruby does have support for callcc, which most lisps, specially the fast ones
like Common Lisp, don't, since it is a pain to implement a fast one without
implications in your whole language. Can't comment on smalltalk, since I never
programmed in it.

~~~
gngeal
_Ruby does have support for callcc, which most lisps, specially the fast ones
like Common Lisp, don 't, since it is a pain to implement a fast one without
implications in your whole language._

Gambit-C is one of the fastest Lisp implementations around, and it _does_ have
a full and performant implementations of call/cc. And I'm afraid that Common
Lisp happens to be "fast by sccident" rather than "fast by design" \- there's
probably more extra speed to be gained by having a better implementable
standard (which CL isn't, or at least not that much) than by omitting call/cc.

(Also note that call/cc is sort of out of favour these days, delimited
continuations is where it's at.)

~~~
lispm
Well, actually speed was a design goal for Common Lisp. Thus the standard
defines the semantics of a compiler, including a file compiler. The CL
standard provides stuff like stack allocation, inlining, fast function
calling, compiler macros, type system, type declarations, compiler strategy
annotations, lots of primitive constructs (like GO in TAGBODY). Most real
implementations add several more speed-relevant features.

If you say that Common Lisp 'happens to be fast by accident', then this is
completely misleading. 'fast' was a design goal and many implementors have
done quite a lot to make demanding applications (like huge object-oriented CAD
systems) to be fast.

~~~
gngeal
_Well, actually speed was a design goal for Common Lisp._

It might have been one of the design goals, but it wasn't the ultimate design
goal (which, I believe, was making a convergent compromise dialect to rule
them all), otherwise certain parts of the language would have been most likely
simpler. The things you mention are the good stuff. The problem is that
there's also some bad stuff that hindered performance of CL on stock hardware
until quality compilers were written. Yes, SBCL is quite fast. But you'd have
a hard time trying to convince me that even modern implementations of Common
Lisp represent some sort of performance optimum in the same way that, say,
Stalin Scheme represents an optimum.

Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail recursion
by structuring the "TAGBODY basic blocks" into mutually recursive function
calls? The same argument could be applied to inlining: Yes, the Common Lisp
standard mentions inlining. But did you notice how nowadays, that's more or
less considered an implementation issue rather than a language issue? How JVMs
are capable of inlining just fine without having to be hinted what to inline
and what not to inline?

Now I'd understand some of the hints, perhaps DYNAMIC-EXTENT makes somewhat
more sense if the purpose is to say "It's my intent to create this value for a
limited period of time, yell at me if I pass it out", but in general, from
where I stand, a lot of the design decisions seem to make writing Common Lisp
compilers actually more complicated to modern compiler writers.

~~~
lispm
Common Lisp had several design goals. Speed was an important design goal. The
users of Common Lisp wanted to be able to deploy and run large and demanding
software on stock hardware.

> ultimate design goal (which, I believe, was making a convergent compromise
> dialect to rule them all), otherwise certain parts of the language would
> have been most likely simpler.

No, there was no ultimate design goal and Common Lisp was not designed as a
compromise dialect at all. Common Lisp was designed to be a modern replacement
for Maclisp and was incorporating several projects working on a modern Maclisp
variant, for example NIL, Spicelisp and Zetalisp. Common Lisp was not designed
to compromise over Emacs Lisp, Franz Lisp, Portable Standard Lisp, Lelisp, UCI
Lisp, Interlisp, Scheme and whatever dialect was there at that time (Common
Lisp first round of design was between 1982 and 1984. Then the standardization
of ANSI CL was worked on until the early 90s.) If Common Lisp was compromise,
then mostly between Maclisp successors. Later during standardization the
Condition System of ZetaLisp was integrated, the LOOP of Maclisp and CLOS was
developed as a new object system (based on New Flavors from Zetalisp and
Portable Common Loops from Xerox). CLOS was also not as a compromise, but a
radically new system optionally based on a Meta-Object System.

CLtL1 says: Common Lisp is a new dialect of Lisp, a successor to Maclisp.
Influenced strongly by ZetaLisp and to some extent by Scheme and Interlisp.

> that hindered performance of CL on stock hardware until quality compilers
> were written

Quality compilers were written almost from day one: CMUCL, Allegro CL,
LispWorks, Lucid CL. Of those Lucid CL was the fastest with the most
optimizations.

Note that there are two or three different types of speed that needs to be
addressed in CL: 1) the speed of unoptimized, fully dynamic and safe Lisp. 2)
the speed of optimized, non-dynamic, and possibly unsafe Lisp. 3) the speed of
production applications which need to be safe, somewhat dynamic, but
optimized.

> Also, wouldn't TAGBODY/GO be replaceable in presence of proper tail
> recursion by structuring the "TAGBODY basic blocks" into mutually recursive
> function calls?

TCO is done by some implementations. Generally it is not seen as a useful
default feature. Common Lisp favors iteration over TCO-based recursion. That's
also what I favor. Since there are some implementation targets which don't
support TCO (or where TCO makes implementations more difficult, one went to
the simpler language). For example some of the Lisp Machines at that time did
not support TCO. Today for example the JVM (where Common Lisp also runs on)
does not support TCO. Scheme is seen as a different Lisp dialect and CL mostly
learned from Scheme the default lexical binding - not more. Remember Scheme is
much older than CL. CL did not adopt Scheme's one namespace, its calling
semantics and conventions, not its naming conventions, not CALL/CC, not its
macro system, ... This again shows that CL is not a compromise dialect - it
simply does not support a lot Scheme-style programming - even though Scheme is
a decade older than CL. By design.

> Yes, the Common Lisp standard mentions inlining. But did you notice how
> nowadays, that's more or less considered an implementation issue rather than
> a language issue? How JVMs are capable of inlining just fine without having
> to be hinted what to inline and what not to inline?

Common Lisp was not designed for one special implementation platform. Some CL
compilers may honor INLINE declarations (many do), others do inlining
automatically (like Allegro CL). If an implementation of Common Lisp on the
JVM uses its inlining capabilities, fine. But there are many other
implementations. ECL for example compiles to C and nobody is would say it is a
good thing to stop working on that, just because the JVM exists. The JVM is a
language issue. It's the Java Virtual Machine. It was not designed to host
Lisp or support efficient implementations of Lisp. One can implement Lisp on
the JVM, but it is a not so good fit.

> a lot of the design decisions seem to make writing Common Lisp compilers
> actually more complicated to modern compiler writers.

You are vague on these issues. Sure a good Common Lisp compiler is complex,
but because the complexity is in things like the type system, type
inferencing, etc etc. The base language itself is relatively simple. The
result is that CL compilers still are producing faster code than most dynamic
languages. Native implementations are still much faster and tighter than what
the JVM can offer.

------
stephth
What is the DSL used in the Ruby example?

------
drcube
What's wrong with distributing source + makefile?

------
_sabe_
I think todays programmers is doing everything back ways. First they think
about what language they want to use, then what libraries do they need, and
then finally on what platform will it run and how will it be deployed.

In the old days everything was much easier. First you chose a platform,
depending on budget, support and so on. When you decided what platform to use,
that then dictated what libraries you had at hands, and what package manager
you needed for deployment.

That maybe sounds very limiting, but on the other hand you don't always have
to come up with new solutions how to bundle and deploy stuff all the time...

~~~
FooBarWidget
Programmers "today"? Maybe I'm not old enough but I cannot remember a time
when programmers would choose a platform based on their target userbase,
unless they are forced to do so (e.g. iOS). Even back in 2000, given the
choice, programmers would always choose to use their favorite language if they
can. I remember that back in the days I was very much concerned about
distribution. If I write an app in Java, users needed to download a 20 MB
runtime (mind you, circa 2000 dialup was common). Visual Basic, Visual C++,
etc all required some form of runtime. .NET was in its infancy and not widely
distributed. I chose to write as much as possible in Delphi so that users only
had 1 executable to worry about, even when Delphi wasn't the most productive
platform. I asked questions on various developer forums, and I was being
ridiculed for thinking so much about user distribution; the consensus from the
community was: pick a language they like, and the user will just have to put
up with whatever additional multi-megabyte runtime it requires.

~~~
_sabe_
Back in 2000? Good tools like Make where built in the 70s that's utilized by
Autotools and other great build systems to bridge the gap between different
platforms, so that you don't have to distribute as the author of the article
we're discussing, a whole runtime with all dependencies.

~~~
TeMPOraL
But the main point of the article was (which I strongly agree with) that
running stuff like Autotools, gems, or package manager kind of sucks when you
just want to install a console tool.

At least back in the "old times" we were distributing program + additional
runtime, and not requiring a random unexpected runtime in proper version +
whole supporting infrastructure + program source code to build. The former
option greatly simplified installation.

