
Time to hand over the reins before Capistrano costs me my youth? - codebeaker
https://groups.google.com/forum/#!topic/capistrano/nmMaqWR1z84
======
forsaken
I just wanted to point out how poisonous our community is. It's something that
I've been struggling with for a long time, and trying to slowly change.

The fact that people read this article, and don't feel the need to mention his
fear of releasing software just shows how broken things are. It shouldn't be
an accepted fact of open source that if you release new code that might be
backwards incompatible, you get vitriol for it.

His quote:

... but I too cowardly to release it and make it mainstream, as Im afraid
it'll destroy whatever good will for open source I have left when the flood of
support questions inevitably comes in, followed by all the people who are
unhappy with what I've built and feel obliged to tell me how bad I am at
software.

~~~
daenz
I'm the author of a popular open-source project for my programming-language-
of-choice. Before I launched it ~3 years ago, I reached out to a well-known
OSS guy in the community. You would probably know his name.

I wrote to him asking if he had any feedback on my project, since I'm about to
release it, and because he had a similar project that didn't get much
traction. He didn't reply...fair enough, I'm sure he's a very busy guy.

Fast forward a few months, I launch the project and it blows up on HN for a
few days, and on twitter for a few weeks. The majority of people love it! But
sifting through the feedback online, I'm shocked and disappointed to see is
the very person I reached out to _trashing_ my project publicly in comment
sections and on his twitter feed. No constructive feedback, only how the
project itself was a horrible idea, and the code was disgusting. I was
dumbfounded...before, I wasn't worth 2 minutes to reply to in an email, but
now I am worth destructive criticism on social media by the same person?

I waited a few months and sent another email asking for advice on maintaining
the project. No response still. Yet every now and then, to this day, I still
see backhanded comments by him about my project whenever someone mentions it
to him on twitter.

Some people are toxic, petty, and childish. I don't really have a lesson from
this, but that's my little story.

~~~
ajsharp
Some people are just trolls, better known as haters. Don't let em get you down
:)

~~~
Dylan16807
...those are not the same thing.

------
patio11
Thanks for creating software which has been an immense service to the
community, and which I rely on quite a bit.

Tangent mode on:

Somebody really, really needs to write the How To Deploy Rails Without Losing
Your Sanity handbook. I will buy a copy. It will sell thousands of them.

A lot of the problems with people's interactions with Capistrano are
environment/ops problems which have known solutions that work, but which rely
on people having a great understanding of arcane trivia which is spread across
conference presentations, blog posts, commit messages, and the practical
experience of the best Rails teams. Unless you're prepared for an
archaeological expedition every time you start a new Rails project, you're
going to do something wrong. You should see the bubblegum and duct tape which
I came up with, and it mostly works, but I know it is bubblegum and duct tape.

Example:

 _Non-deterministic deploys of code from (usually) un-tagged source control_

I feel lucky in that I was mentored by an engineer who decided to teach me,
one day, Why We Tag Shit. But for the Why We Tag Shit discussion, I would be
like almost every other intermediate Rails engineer, and be totally ignorant
of why that was a best practice until lack of it bit me in the keister, at
which point the server is down and one has to rearchitect major parts of the
deployment workflow to do things the right way. Why We Tag Shit is only about
a 500 word discussion, but it's one piece of organic knowledge of the hundreds
you need to do things right, and it is (to the best of my knowledge) not
covered in docs/QuickStarts/etc because that seems to be out of the purview of
the framework proper (I guess?).

I'm sure that I'm ignorant of several of the hundreds of pieces of things one
needs to do to do deployment right, as evidenced by my fear every time I
execute my deploy scripts. I, and I must assume many other companies, am
willing to pay for an option which gets me to a non-bubblegum and duct tape
outcome.

Seriously, folks: there is a product here.

~~~
raverbashing
I'll be downvoted by the "newer generation" but here's a pet peeve of mine

The same reason "why we tag shit" is the reason downloading packages from
gem/pip _is not enough_

Who guarantees that when you do your deploy, the library you need is there?
How many times did your build break because it had a glitch downloading the
package?

Keep a way of rebuilding and deploying your software. _All of it_

~~~
lamby
Mmm. I inherited a rather hardcore strain of liking entirely deterministic
builds from being a Debian Developer and would never give up the dependability
of it now. In fact, the thought of a build process connecting to the internet
to download code just seems so utterly broken in my world view I actually find
it difficult to explain it properly to others.

I'd love to be able to do it without embedding code copies, or at least have a
smarter system than that.

~~~
Volpe
But for the most part these build processes are for web apps... If your build
can't connect to the internet, your web app isn't going to work (even if it
can build).

~~~
nirvdrum
Sure, but it's not like the entire Internet is up at once. If rubygems.org
goes off-line, it shouldn't mean you can't deploy your Rails app.

------
codebeaker
I'm the OP of the mailing list post, and have maintained Capistrano for the
last 5 years. I'm passionate about providing great open source tools, my
business and reputation are built on Capistrano and I don't want to give it
up, but it's destroying me.

~~~
sandGorgon
Kickstarter ?

I mean for something as operationally critical like Capistrano, why dont you
seriously raise money on Kickstarter, outsource some of the testcases, etc.
and be more productive overall.

And please feel free to raise money to fix RubyGems as well ;)

~~~
palidanx
I don't know how this would vibe in the open source community, but I'd
definitely donate to help out a Capistrano project.

------
bretthopper
Some unsolicited advice from someone who's never run an open source project as
popular as Capistrano:

* Ditch v2 ASAP (seems like you've already decided on this). It's pretty obvious you aren't motivated to work on that codebase anymore. I've looked at v3 and it's much better thanks to relying on Rake tasks.

* Be selfish. It's your project so if you think v3 is the way to go forward, go with it and who cares what the "community" thinks.

* Seems like you already have a few people helping out, so continue and maybe make formal "core" team. There's nothing with yourself taking a step back from the heavy coding. But I believe that Capistrano would be better with your guidance than without it.

codebeaker: There was no mention of Harrow in that post. Are you still working
on that? I'd assume that if you were you'd continue work on Capistrano since
it's based on it.

~~~
smacktoward
_> Be selfish. It's your project so if you think v3 is the way to go forward,
go with it and who cares what the "community" thinks._

Or call the community's bluff. "I think v3 is the way forward, so that's what
I'm going to be working on. If you want to stick with v2, _you_ maintain it."

~~~
FooBarWidget
And then prepare to be getting accused of elitism, being selfish, not caring
about users, crippling their system, etc. For a good example, see how people
treat Lennart (author of systemd).

~~~
desas
and the author of pulseaudio and avahi. He's taken a lot of abuse over the
years.

------
gnufied
For real long time Capistrano v2 has been exclusively going forward with Pull
requests and next to no new development while Lee worked on v3 on separate
branch, which looks like a rewrite.

As a result various releases of v2 were buggy. Capistrano is a hard to test
application agreed but its test coverage is plainly woeful.

About 6 months back when 2.4.12 release was broken
([https://github.com/capistrano/capistrano/issues/434](https://github.com/capistrano/capistrano/issues/434))
I suggested to remove asset pre-compilation stuff from Capistrano. Capistrano
is a general purpose tool, company where I work we use it for deploying java,
php, ruby and all sort of stuff. I don't understand why it should have poorly
tested asset pre-compilation things built in.

I don't know what made Lee work on a rewrite. I can only imagine how difficult
it must have been for him to work on something so big singlehandedly while
running a company.

His last point is very valid about using RVM, rbenv etc in production. I don't
know why people do that. Does that make it easier? Aren't people aware of
something like - [https://launchpad.net/~brightbox/+archive/ruby-
ng](https://launchpad.net/~brightbox/+archive/ruby-ng) ?

~~~
cheald
The reason I use RVM in production is that it makes it trivially easy to:

1) Use multiple parallel Ruby versions for transitions and rollbacks - for
example, when we upgraded from Ruby 1.9 to 2.0, we installed 2.0 via RVM, ran
our staging environment on 2.0 while production ran on 1.9. The instant-
rollback safety net is great, because if we run into some obscure bug on one
VM, we just change our RVM environment config var and deploy and we're back to
known-good in seconds with no downtime.

2) Run applications that require different Ruby version levels. For example,
Puppet 2.7 wasn't compatible with Ruby 1.9+, but our app was running on 1.9.
With a single system install, we'd not be able to run both.

The point about stepping out-of-bounds in regards to LTS distros is extremely
valid, but at least in our case, it's not about "just effin' compile
already!", but rather about flexibility and resilience - by using RVM, we
absorb the burden for making sure things work rather than delegating it to our
distro, but it's worth it for the things we pick up.

~~~
jeltz
I would recommend running using chruby or just ruby-install and $PATH (chruby
just changes the path variable) for those things since it is less invasive.

Personally I always try to run ftom apt in production.

~~~
regularfry
+1 to this. I wrote up how to build a ruby .deb from ruby-install here:
[http://www.blackkettle.org/blog/2013/08/26/how-i-ruby-
part-2...](http://www.blackkettle.org/blog/2013/08/26/how-i-ruby-
part-2a-deployment/)

------
AlexMuir
My first thought was "I owe this guy, Capistrano is the main reason why I have
spent ~ $100 per year on VPS servers and not $100 per MONTH on Heroku et al.

I'd suggest Lee runs a Kickstarter type thing and I'd very happily throw in
$100. But I don't think he will because it doesn't seem quite right.

So here's a (wild and completely off the cuff) startup idea - a pre-emptive
Kickstarter. Someone creates the project "Lee Hambley, continue working on
Capistrano." and we all pledge into the pot. If Lee agrees to do it, he gets
the money. If not, we don't pay anything.

~~~
forsaken
Or just give them money on something like gittip. Things like this need to be
sustainable, not just a flash in the pan of $10k, then going back to having
nothing.

------
joevandyk
Really looking forward to Docker being 1.0.

What you want to do is build a single package of everything your application
needs (which includes the application code and _all_ dependencies -- libc and
up), then copy that package to the production servers.

It shouldn't matter if your application server has Ruby 1.9.3 and you need
2.0.

It shouldn't matter if the last deploy of your app needs Nokogiri compiled
against libxml 2.8 and you now need 2.9.

It shouldn't matter if you are running 5 different apps with 5 completely
different set of dependencies on the same machine.

It shouldn't matter if you need to use the asset pipeline.

It shouldn't matter if github or rubygems drops out half-way through the
deploy process.

All the production server should get is a single package of all that your
application needs, then a 'restart application' command.

Docker should be able to handle all this simply.

------
alrs
As always, it bears repeating: rvm/rbenv don't belong in production. They
exist to allow developers on Macbooks to sync their version of Ruby with
whatever is packaged in the Linux distro or BSD variant that runs in
production.

If I had a Mac I'd skip the ad-hoc Ruby environment switchers and skip
straight to Vagrant.

~~~
3pt14159
I used to agree with you. Why would you take a 20 or 30 percent performance
hit when you are busy shelling out tens of thousands for hardware load
balancers.

Then one day a rare instruction set on our colocated server (so not something
super standard like linode) was specified during the compilation step of ruby
and (due to an extremely subtle co-bug between ruby and the compiler). It took
us fucking weeks to find this hisenbug that was somehow causing workers to
drop, but only during times of very high load.

Probably lost $500k worth of customers, dev time, and company moral.

Now I have a different view. Keep things as simple and as "normal" as
possible. That way you can always upgrade to the next version, you don't hit
weird bugs when you libraries assume that Time.now is second-accurate, instead
of sub-second accurate (MRI vs Enterprise Ruby).

RVM is made for production
([http://stackoverflow.com/a/6282260/384700](http://stackoverflow.com/a/6282260/384700))
and it saves a lot of headaches to just go with the flow. As for mac dev, I
agree that it is a waste of time compared to working out of Ubuntu, but
designers like photoshop and Vagrant is non-trivial for them to set up,
especially for people that work on multiple projects.

~~~
otterley
Pointing to the author's declaration that "rvm was made for production" is not
exactly the best evidence of such, especially in light of mine and my
colleagues' endless hours trying to figure out how to fit it into a non-
interactive-shell environment.

~~~
eulerphi
I've had similar problems with RVM. Ultimately I used the system-wide RVM
installation and this chunk of code:

    
    
      export PATH=$PATH:/usr/local/rvm/bin/rvm
      . /usr/local/rvm/scripts/rvm
      . $REPO/.rvmrc
      . $(rvm jruby-$JRUBY_VERSION do rvm env --path)
      cd $REPO
      bundle exec torquebox deploy
    

Definitely not ideal and I spent a large amount of hours figuring out how to
use it non-interactively. I'm still somewhat worried that some things might go
wrong if some of the code I didn't write in this repo attempts to call
binaries directly. Basically with bundle exec --deployment, all gems are
stored in $CODE/.vendor which is necessary to allow users to install their own
gems, with the root installation. Bundle exec has to be used, I messed around
with RVM wrappers, but they don't work with the --deployment bundler use.

~~~
otterley
You're doing too much. All you need to do is:

    
    
      source /usr/local/rvm/environments/$BUILD
    

You can put it in a wrapper script if you like. It will unset all conflicting
environment variables first.

~~~
eulerphi
The thing is that the rvmrc has the jruby version env set. So I need to load
it first. And it contains an 'rvm use' which I don't want to fail. I suppose I
can slim the rvmrc down to just include the environment and then all I would
need to do is source the rvmrc? Thoughts?

~~~
otterley
I don't believe you need to source the rvmrc. The line I pasted above does all
the necessary things that "rvm use $BUILD" would do for a shell, without all
the things that require an interactive shell.

~~~
eulerphi
The rvmrc is part of the repo and contains the java_opts env.

------
AhtiK
Does anyone know what's wrong with the Rails asset pipeline that is mentioned
in the post as one of the issues?

~~~
codebeaker
There's a number of problems, but foremost is that there's no good way to
"roll back" assets, and there's no concept of keeping assets that might be
used by old versions of pages cached in CDNs, when they have been replaced by
newer assets. This is a problem of the manifest system, and of the `assets`
directory always representing the current newest state, not the collective
state since the beginning of time. Maintaining state from the beginning of
time would bring it's own problems, thus many of the workarounds about
tracking old, and new assets are time consuming and sub optimal, and
unfortunatley people need them. There's a cap task which touches `mtimes` of
all referenced assets, which can typically take 5 minutes to complete. It's
naïve, and stupid, but it's the only solution (that we could come up with) to
a real asset pipeline problem.

I'm also of the opinion that compiling assets in production as a part of your
deployment process is insane, there's so much magic in the Rails asset
pipeline that it's not uncommon to turn up bugs where tables don't exist, and
the rails app can't initialize, or some javascript runtime environment isn't
found which can leave your deployment in a broken state.

I'm firmly of the opinion that assets should be compiled and checked in, but
then of course you run into problems with rails serving those in development
mode, rather than the development files.

All these issues are fixable, but they're all indicative of tools that aren't
quite mature yet, and as Capistrano sits on the boundary of where these
problems come to light, it seems to fall to us to deal with it, and to educate
people on what they ought to be doing.

Education is no problem, I really believe that the de-facto standardisation of
Rails-like deploys (i.e timestamped releases, with common linked directories,
and a symlink to the current active timestamp) is an excellent result for
knowing what to expect in an environment where there's hundreds of ways to get
Rails apps running, but it's still not as smooth a process as it could be.

I'm familiar with at least one project that's been re-written in Scala and
Java because the previous version was prohibitively difficult to deploy as it
was in RoR. (GrayLog2, to namedrop them)

~~~
irahul
> I'm also of the opinion that compiling assets in production as a part of
> your deployment process is insane, there's so much magic in the Rails asset
> pipeline that it's not uncommon to turn up bugs where tables don't exist,
> and the rails app can't initialize, or some javascript runtime environment
> isn't found which can leave your deployment in a broken state.

This is one of the pain points for me. Even when everything is setup, the
asset compilation churns the disks and takes way too much cpu. I would rather
do that on my local machine than disrupt the production server. I do a simple
workaround for that:

    
    
        namespace :deploy do
          namespace :assets do
            desc 'Run the precompile task locally and rsync to shared'
            task :precompile, :roles => :web, :except => { :no_release => true } do
              run_locally "rake assets:precompile"
              run_locally 'rsync -e "ssh -i production.pem" --recursive --times --rsh=ssh --compress --human-readable --progress public/assets #{user}@#{host}:#{shared_path}'
              run_locally "rake assets:clean"
            end
          end
        end

~~~
mnutt
My deployment process does this too. One thing to be careful of is to make
sure you're not deploying a different revision than the one you are on
locally. For instance, if you have uncommitted changes locally, they will be
reflected in your assets but not in the rest of your deploy.

~~~
irahul
Yes. Different branch and uncommitted changes are an issue. Different branch
can be solved by switching the branch if `git rev-parse HEAD != git rev-parse
branch`. Uncommitted changes can be handled by `[[ -n $(git status
-s)]]`(simply abort).

------
tomdefi
For anyone interested in an overview of Capistrano v3, I wrote an introduction
last week -
[https://medium.com/p/ba896a142ac](https://medium.com/p/ba896a142ac)

~~~
codebeaker
Tom's article is a really great run down of all the awesome things that we
built.

------
ealexhudson
A good decision - get out while things are still positive. Not enough people
are brave enough to step down at the right time (or even when it's obvious
it's already the wrong time).

------
kawsper
I am a bit sad that he feels this way about it.

I have used Capistrano a lot, I built my "default" setup, compiled it into a
gem, and released it here: [https://github.com/kaspergrubbe/simple-capistrano-
unicorn](https://github.com/kaspergrubbe/simple-capistrano-unicorn) and moved
on with my life as a developer.

I know of at least two bigger organizations that depend on Capistrano (and my
gem) for their deploys. I feel like Capistrano is the way to go if you manage
your own servers, and need to deploy to them.

Capistrano started my Rails experience, and I am very grateful for the work
put into it. But I never wrote and said "Thank you" or "Great job", maybe we
need to be more vocal to the people that put in time and energy to build the
software that we use a lot.

------
joeblau
It's sad to see when an open source project becomes overwhelming. On one hand
the project is open source, so hopefully, someone else can pick up the torch.
We saw this happen in the node.js community and node's been moving along. On
the other hand, based on what Lee is saying, it looks like situation is pretty
bleak. I'm not a Rails user, but I feel like most of the "hot-startups" in San
Francisco run a Ruby stack. From an observer looking into the community and
platform though this post, I never realized how many challenges there were in
that development environment.

------
ChikkaChiChi
As much as this is an open invitation to rail on the RoR community, I think
this is a problem that is a lot more indicative of this brave new software
culture both open source and (independent) commercial.

If your tool sees any sort of uptake, it suddenly no longer is yours. The
community suddenly expects you to not only to continue to modify the base code
to improve functionality, but to also adhere to a sort of backwards
compatability so that everything they know and loved about your baby never
changes.

I can't imagine how much more taxing this would be once the tools you built
become integral part of other team's workflow. The burden and stresses of
keeping "the world" afloat would cause many a sleepness night for people of
strong constitution.

------
joaomsa
Capistrano really has saved us multiple times, sad that a vocal part of the
community tends to exhibit such behavior.

At our company, we develop multiple RoR apps and we've run into many of these
issues (mostly related to the asset pipeline), yet none of them actual
problems with Capistrano. Since it's the bridge between so many things, I can
imagine why it's easy for it to become cannon fodder.

We've tried to standardize many of our recipes such as local asset
precompilation into a single cohesive gem
([https://github.com/innvent/matross](https://github.com/innvent/matross)).
That has saved us the trouble of debugging the same issues over and over when
they inevitably pop up across applications.

------
machbio
Thanks for the awesome software.. I just started learning about capistrano
recemtly, just amazed by how simple it is..

I believe when you said that PAAS will go, only reason I use heroku and
dokku(from docker) is due to its easy deployment.. and for no other reason
than deployment..

------
grandalf
Check out fabric as a much faster alternative to Capistrano. Combined w
cuisine.py it's a simple and powerful alternative to chef solo.

------
chrismealy
I love ruby and rails, but yeah, I'd switch to any framework in any language
that made deployment stress-free. Except php.

~~~
keypusher
Python with virtualenv and pip has been fairly stress free in my experience.

~~~
kawsper
I feel Python apps is quite a pain to install. I had to deal with a Graphite
installation, and it consists of 50% packages from apt-get and 50% of packages
from easy_install.

[https://gist.github.com/kaspergrubbe/5792356](https://gist.github.com/kaspergrubbe/5792356)

Is this an issue with Graphite? Because I feel this setup is quite elaborate
compared to using Rubygems and bundler.

~~~
gingerlime
From experience with both python/django and ruby/rails, I think python is
generally simpler. Start with the fact that python itself is usually pre-
installed on most distros, and usually a recent-enough version to get you
started. Ruby on the other hand is much harder to just get installed, choosing
the right (minor)version, rvm/rbenv choices etc. I tend to compile my ruby,
but it's a lengthy and rather fragile process.

Graphite is both a good and bad example. Good because it is really complex and
documentation is a little sketchy. I've written a fabric script[1] that
automates the process, and it's far from trivial. Bad example, because it's
not really a single app, but a system - a collection of tools with
dependencies. Even if we discount the web server (nginx or apache?), it
includes things like the core "database" (whisper), the event listener
(carbon, which in itself is complex depending on your setup), the graphic and
processing libraries, and then graphite which is a pretty involved django app
with its own sub-components.

So when you say graphite, it's really a full-blown system with lots of moving
parts that need to fit in together. I can't think of an equivalent example in
the rails world, but any rails app with a db, caching layer, and a few other
external components won't be much easier to get set up and running.

[1][https://github.com/gingerlime/graphite-
fabric](https://github.com/gingerlime/graphite-fabric)

------
yannk
"Whilst I believe strongly in Capistrano as a general purpose tool [...] I do
think the future of software deployment is in small, containerised VMs and so-
called PaaS, as what we're all doing right now has to end, some time."

Kudos. It takes a lot of courage to admit your baby is not going to fulfill
the future you had initially imagined.

------
stevewilhelm
Check out 'Deploying Ruby Applications to AWS Elastic Beanstalk with Git' [1]

[1] [http://ruby.awsblog.com/post/Tx2AK2MFX0QHRIO/Deploying-
Ruby-...](http://ruby.awsblog.com/post/Tx2AK2MFX0QHRIO/Deploying-Ruby-
Applications-to-AWS-Elastic-Beanstalk-with-Git)

