
If you love automation, why is your development environment manual? - timruffles
http://sidekicksrc.com/post/if-you-love-automation-why-is-your-development-environment-manual/
======
qwerta
>If you love automation why is your development environment manual?

Because I usually do it once in 5 years :-)

My setup: Linux, Kubuntu LTS. Entire system is backed-up with rsync, so I have
daily snapshots for last 6 months. There is also weekly backup to cloud and
external hdd.

I have primary 250GB SSD which usually sits in my workstation. Whan I travel I
physically move SSD to laptop (takes 10 minutes).

If something went wrong and my primary SSD would die: I just boot from Livecd,
copy 200GB, setup bootloader and reboot. No need to 'reinstall' system from
scratch.

Every six months I take one afternoon and update all my tools (IDE, OS,
apps...). I dont use automated updates except critical stuff such as browser.

~~~
fideloper
I think an important point in this post was more about not trying to setup a
dev environment in your base Mac install (or similar). Use Virtual Machines.

~~~
qwerta
In that case I just find it better to wipe MacOS and replace it with Linux.

~~~
gtaylor
It's not always a good solution for your entire team. I run Linux on my
desktop, but have found that the touchpad drivers (even the experimental ones)
just really suck on Linux+Macbook Pro to the point where it's either external
mouse (not always possible) or decreased productivity.

I have a feeling this is going to devolve into a "well, just use vim and a
tiling WM or X other thing", but the point is that just replacing Mac OS with
Linux isn't always going to be good for your whole team.

~~~
beatgammit
When you're targeting Linux, it's easier in the long-run to just use Linux.
The base system isn't the same between the two because Mac OS X uses BSD
versions of important things like SED, grep, etc. There are subtle differences
that most Mac OS X users I've met don't know and can lead to weird problems
like "why doesn't X work; it works fine on my mac!!".

If everyone's on Linux, the dev and productions systems are more similar, so
less scripting issues can arise. I personally use Arch, other coworkers use
Mint or Ubuntu, and only one developer uses Mac OS X (others switched because
of above problems; he couldn't switch because of EFI problems). Even devs new
to Linux (from Windows) love Mint, and all is well and good in the world.
Servers run Debian or Ubuntu, so I test my changes in a VM because Arch has
noticeably newer software in the repos, but that's because I can't stand
Debian as a desktop system (package manager too slow).

Also, the Mac OS X user uses his Mac like a Linux box. He only ever touches
his trackpad to use non-keyboard friendly apps (uses Vimium for Chromium,
tmux/vim for terminal/editing, etc). Our devs not familiar with the keyboard
generally use Sublime on Linux, which is a fantastic editor as GUI editors go.

The trackpad drivers aren't that bad, but may require some configuration in
the xorg conf (I got multitouch working, but disabled it because I find it
annoying).

We don't standardize on a single dev platform because each developer is
different. The developer understands that using Linux is going to be easier in
the long-run, but he/she must understand that his/her choice of OS must not
negatively impact productivity. We've had a few developers try starting on
Windows, but we'd refuse to support any dev environment besides Ubuntu Linux,
and when they ran into problems, they switched to Mint/Ubuntu.

I've never seen a developer be more productive on Mac OS X than Linux. In
fact, those gesture usually end up slowing him/her down from what should be
easy using the keyboard (and all that switching from trackpad to keyboard is
_not_ productive).

~~~
hosh
There are advantages to using VMs even if you've installed Linux as the base
OS on Mac hardware.

The VMs provide isolation for testing and developing the server components.
(I'm assuming you are developing server code). Through synced folders, you can
still use your favorite $EDITOR on the host machine -- vim, emacs, sublime
text, whatever. Onboarding is faster, and you can manage multiple node setups.

Further, you have the additional advantage of being able to replicate staging
or production systems on your local box. This reduces time spent round-
tripping to a staging server.

This lets you standardize the server code without requiring all developers to
standardize on editors/ide/etc.

Personally, I am more productive on OSX than on Linux. But that's because my
actual dev environment is tmux+vim. I let OSX and Mac hardware handle the
stuff that it does much better than Linux: being able to talk to anyone,
anywhere, with anything. (Networking) Being able to reliably suspend and wake.

------
campbellmorgan
I think the general idea of running a vm with all your dev setup in is great,
but I've often found the small niggles (such as speed of switching quickly
between browser and vm terminal, the fact you need a fairly good computer to
run even quite a modest server with a gui and if you don't use a gui, you're
limited to 256 colours) all a bit more trouble than the benefits.

My solution is to set up my perfect dev environment in linux (I use Fedora) on
a partitioned usb pen drive. Then either boot from it (if i'm on a crappy
computer) or boot a vm direct from the usb if the computer is more competent.
It's also quite nice if there's an emergency and you're away from your
computer -- you can just rush into an internet cafe, plead/bribe them to let
you boot from your usb then have everything, ssh keys inlcuded ready for
urgent repairs!

~~~
scaryclam
I think the point here is that it took you time to set up that dev
environment.

It shouldn't matter _where_ your environment is (a VM, a laptop, etc), you
should be able to recreate it easily and reliably. This is what the automation
tools are for. I prefer salt stack and have a salt config for my work laptop.
If something happens to the laptop or I get a new one I just need to run salt,
wonder off for an hour and when I get back my entire environment is set up. I
also use it with vagrant so that when a new dev joins the team they can run
one command and have a copy of the project setup in a vagrant box, databases
and all, ready to go.

You could set up a similar thing so that if you lose that USB key you can have
your prefect setup reinstalled with minimal fuss.

~~~
pestaa
Would you be able to share at least some of your current salt config? This is
one idea I've been thinking a lot about lately, but never managed to get
beyond the obvious package names, which is trivially installable with one
command on a good Linux distro.

------
kstrauser
I don't want dev to be exactly like stage and prod.

Yep, you read that right. I deploy to Debian servers but develop on Linux Mint
or OS X. The key is that the environments are very _similar_ but not
_identical_. The payoff comes when you break yourself of the habit of relying
on accidents of deployment instead of building a general-case solution that
works in multiple environments.

You scan /proc to look at running processes? That's great, until you're on a
machine without /proc. Better to spend an hour learning how your dev platform
(Python, Java, whatever) abstracts that away for you. Trying to send an email
by shelling out to /usr/bin/sendmail? Oops! That's broken in lots of places;
better learn how your dev platform handles it!

The big win comes when you upgrade your stage and prod environments to a newer
distro - or a different one altogether - and your stuff keeps working because
you'd relentlessly whittled away all the OS-specific dependencies.

~~~
snowwrestler
This strikes me as an example of an unnecessary/premature optimization. Unless
you know that your product will need to deploy on an unknown variety of server
environments, there's no advantage in taking the extra time to 100% abstract
away the server environment.

And really, how trustworthy is that abstraction? You've basically outsourced
responsibility for your server environment to your language community. What if
they create a bug that is exposed on Debian, but not on OS X?

~~~
gtaylor
Yeah, for certain things your abstractions can either add a good bit of
overhead or increase the amount of code you need to maintain. No real point
unless your product really needs to run in a mixed environment in prod.

------
dugmartin
Your don't need to give up your GUI editor to do this. Vagrant sets up a
shared folder automatically in the virtual environment under /vagrant that
maps to the folder where your Vagrantfile is located on the host machine.

I do all my development now using Vagrant running Ubuntu VMs but I still do
all my editing with the same Windows editor I've used for years running on the
host machine.

~~~
npsimons
Or, you know, ssh -X. Oh, but wait, they're trying to get rid of that in
Wayland, "because nobody uses it" . . .

I suppose there's always RDP or VNC, if you can stomach their limitations.

~~~
pekk
That has never been accurate. Wayland has planned X support for a long time,
it just won't be in the core

------
VLM
The author missed a golden opportunity to discuss battery life. Your laptop
will run longer if you SSH/VNC/whatever into the virtualized development
machine if you don't host the virtualized image on the laptop.

Also I can SSH/VNC/Xwindow/whatever into a server machine with performance
stats far beyond any currently imaginable laptop. Its like owning a laptop
from 2023 today in 2013.

~~~
23david
The latency is a pain. I like being able to work from anywhere by tethering my
iphone5's LTE connection.

Easier and more reliable to have everything working locally (With local dev
environments using vagrant/virtualbox) and use the internet connection for
syncing git changes or doing research.

~~~
archivator
There's the Mosh project (<http://mosh.mit.edu>) which seems fit for your use
case. I don't have personal experience with it but it looks promising.

~~~
23david
wow looks very cool. never heard of it...

<http://mosh.mit.edu/#techinfo>

How Mosh works Remote-shell protocols traditionally work by conveying a byte-
stream from the server to the client, to be interpreted by the client's
terminal. (This includes TELNET, RLOGIN, and SSH.) Mosh works differently and
at a different layer. With Mosh, the server and client both maintain a
snapshot of the current screen state. The problem becomes one of state-
synchronization: getting the client to the most recent server-side screen as
efficiently as possible.

This is accomplished using a new protocol called the State Synchronization
Protocol, for which Mosh is the first application. SSP runs over UDP,
synchronizing the state of any object from one host to another. Datagrams are
encrypted and authenticated using AES-128 in OCB mode. While SSP takes care of
the networking protocol, it is the implementation of the object being
synchronized that defines the ultimate semantics of the protocol.

------
druiid
I fully agree with this. 'Before', the dev team used a mix of, well, I'd
rather not say. It was just not very pretty and eventually everyone got their
work done. A while back I built a vagrant+puppet configuration for everyone
and the team has since transitioned to using that.

It's nice to have at least semi-parity with the production environment. This
is possible because we can utilize most (not all, due to no central Puppet
server) of the same Puppet modules which we use (or I built), in the VM.

Essentially what you gain is not worrying about if a developer will break
their machine/VM and go 'Whoops, can you fixor it?'. Additionally, you no
longer have to worry about such things as php/node/ruby version issues with
the dev/production side of things. We've gone from having sometimes issues
with certain code not running the same as in development, to just throwing it
up on the staging environment through deployment tools and it just runs!

------
hleszek
obligatory xkcd: <http://xkcd.com/1205/>

~~~
medecau
"xkcd #1205 says I don't need it."

------
dmytton
We do this using Vagrant and Puppet and actually just blogged about it
yesterday: [http://blog.serverdensity.com/many-projects-with-vagrant-
and...](http://blog.serverdensity.com/many-projects-with-vagrant-and-puppet/)

------
CatMtKing
I've heard of Salt Stack[1], though I've never used it. From what I can tell,
it can be used to do the same thing.

[1] <http://docs.saltstack.com/>

~~~
jsdalton
The Salt Stack team also has a plugin built specifically for provisioning
environments in Vagrant called Salty Vagrant:

<https://github.com/saltstack/salty-vagrant>

(I haven't used it, but I have used -- and love using -- Salt Stack.)

------
roel_v
It's funny that this article proclaims to combat yak shaving while it actually
promotes it.

~~~
hobs
I take your point, but I would say that the article really encourages shaving
the yak one time, instead of using scissors to beat it back occasionally.

------
doki_pen
There is something to be said for running your software on multiple
environments. There is also something to be said for stubbing out pieces of
your stack for testing and development purposes. There is often a lot to be
learned. That said, I've used the VM approach on projects with high turnover.
It is usually not the same as production deployment, so it does take effort,
but depending on the team it might be worth the effort.

------
orclev
I tried this on a recent couple projects and it failed miserably on both of
them for completely different reasons. On one, the architecture of the project
dictated really really fast network connections and the overhead from the VM
choked the project. That's probably an unfair example because this was
actually a toy project that was never meant to run in a serious environment
(hence the requirement of a huge bandwidth virtually 0 latency network
connection). The other failure however is more serious. Our company (and at
least one project) dictates a Windows based development environment, and the
abysmal performance of NTFS combined with a massive project code base meant it
was taking 2+ minutes just to run a status check on the version controlled
code.

I think given a Linux host environment this might work better, or a smaller
code base, but at least in these two situations it was a failure.

~~~
bryanlarsen
Note that the recent release of VMWare support for Vagrant may at least
partially mitigate the problems in your second scenario.

------
markeganfuller
I prefer having a VM built with Vagrant (and in our case puppet) which runs
the same configs as our servers while having my development tools (editors
etc) installed on the local machine. That way all my apps run at native speed,
and the server replicates (almost exactly) the production environment.

------
karlkatzke
Issue: What about VPN? If you're connecting to your corporate network with
OpenVPN, and you are restricted to one connection (common sense security
policy) then you need to get your infrastructure team to issue you a separate
key for each VM that you're running that you want to be able to access a
tunnel and get on your corporate network.

I realize that this isn't an issue for 99.9% of startups, but in the non-
startup corporate world where there are security implications to keeping
everything in a publicly accessible code repo (healthcare, government,
education in some states) then you've got another complication.

The only thing I've been able to work up so far is sharing a folder on my
desktop through to the VM and running commits from my desktop.

~~~
druiid
I would say that you probably shouldn't be doing VPN to your development boxes
in general. I'd see this as a security risk. I generally don't let anything
'production' touch directly anything which might not have the same security
standards as the production gear.

------
hosh
Vagrant + Chef is great!

Vagrant + Chef + Berkshelf is better!

You can define the cookbook dependencies in the Berksfile and version it along
with Vagrantfile. Berkshelf is like bundler for your cookbooks. It assembles
cookbooks from a variety of sources and loads them into your Vagrant-managed
vm. Check it out at <http://berkshelf.com>

To help get things going in that end, I've been working on Kamino
(<https://github.com/hosh/kamino>). It's intended to generate a
Vagrantfile+Berkshelf file for use in your project. Right now, it only
supports a basic Rails stack.

Pull requests welcomed :-)

------
graycat
The first reason is, people who build tools mostly don't think that the tools
will or should be used as components of other tools for more in 'automation'.
The result is that it's tough to build on such tools.

The second, and more specific, reason is that location of one of my most
intense love/hate dilemmas, Xerox PARC that pushed the first 'graphical user
interface' (GUI) instead of what came before, usually typing text into a
command line. Command lines are mostly easy to automate. GUIs are mostly a
total PAIN to automate.

I'm wanting to automate, willing to automate, waiting to automate.

------
cpursley
Cloud development like www.nitrous.io (previously action.io) is the future.
I've played around with it am quite impressed. Basically no setup for standard
platforms like rails and node - choose your dev environment, install your
dependencies if not already there and get coding. It's a Linux environment so
your dev is same or close to production. And the IDE is quite nice. Caveat:
Yes, you have to have a connection. But, for those few precious unconnected
moments, I don't want to be coding anyways.

------
rb2k_
Any comments on what people use to manage the chef cookbooks (and why)?

Librarian or Berkshelf seem to be the two main contenders to make updating
cookbooks similar to updating gems with bundler.

~~~
tomku
I've tried both Librarian and Berkshelf to manage cookbooks for my Vagrant VMs
for use with Vagrant's chef-solo provisioner. They both have nearly identical
core features. Both can grab cookbooks from the Chef community site, local
path, generic Git repository (which includes getting a single cookbook from a
subdir) or Github. Their syntax is nearly identical. The only difference that
I can think of is that Berkshelf has pessimistic version matching support (a
la Bundler), while I'm not sure that librarian-chef does.

Where they differ is in where they put your cookbooks, and what additional
features they offer. Librarian-chef puts your cookbooks in a "cookbooks"
folder in the project root, while Berkshelf keeps a global cache of all the
versions of each cookbook you've installed outside of the project. Both
provide support for uploading your cookbooks to a Chef server, librarian-chef
through plain knife and Berkshelf through it's own upload command.

Berkshelf also features scaffolding support for Vagrantfiles and new
cookbooks, and generally has extra features specifically for rapidly iterating
on cookbooks themselves. It also provides a vagrant plugin to manage
bundling/uploading cookbooks to a Vagrant VM, since they need to be copied out
of the central repo and made available.

Overall, I've found that I like Berkshelf better. It feels a little more
idiomatic, and I like having my cookbook versions shared between projects
rather than duplicated everywhere. If you have any specific questions about
either, I'd be happy to answer them.

~~~
bryanlarsen
I've never tried Berkshelf, didn't even know about it. librarian-chef works
great for our needs; I doubt it's worth investigating alternatives.

I will note that the librarian-chef also has a vagrant plugin: vagrant-
librarian-chef.

~~~
hosh
Berkshelf is worth investigating simply because there's a lot of momentum
behind it, even though it came out after Librarian. By "momentum", I mean that
since the session introducing it en mass at the Opscode Community Summit in
2012, it's being adopted by the community.

No hard numbers, but the main thing is that Berkshelf could not have really
happen without enough standardized community cookbooks. By putting the
cookbooks into a global directory, it works more like Ruby gems and Bundler:
instead of focusing on your customized (and probably divergent) cookbooks
loaded into your project directory, you focus only on the few cookbooks that
are specific for your needs. It comes out of a bigger theme in the 2012
Summit, that of creating "library cookbooks" that gets assembled by your
"application cookbook". If it's a library cookbook, you don't necessarily need
it in your project library any more than you need to import a Ruby gem into
your Rails directory during development.

Test-kitchen is also worth looking into, though as far as I know, that still
uses Librarian under the covers. Berkshelf support for test-kitchen is pending
(<http://tickets.opscode.com/browse/KITCHEN-9>).

In summary: Berkshelf is part of the overall trend in modularizing these
cookbooks and developing tighter engineering discipline.

~~~
bryanlarsen
I guess I'm still confused -- that's pretty much exactly why and how I use
librarian-chef. After all, it's explicitly modelled on Bundler

------
mwcampbell
I've often thought that it would be nice to have a script that rebuilt the
software installation on a developer's workstation from scratch, completely
automatically, using only configuration files and scripts checked into version
control somewhere. Such a process could run every night while the developer
sleeps. This would ensure that it's dead easy to bring a new developer fully
up to speed on a team.

~~~
hosh
I know a couple guys who do that. Their Steam app is also managed by Chef.

We got our dev setup partially automated like that. Even the partial setup was
better than what I heard -- took a week to set up a laptop with the tools and
the whole platform. When I was onboarded, it took about day, more or less
hands off.

------
andrewcooke
if you use an ide you can do something similar, but use the vm client
directly. i do this all the time, and virtualbox's seamless mode means that
you can mix windows from client and host on the same screen (except ubuntu
unity?).

it's particularly good when different clients have different OSs. and you can
even do hardware development - i have tested usb drivers in a vm client that
talk to hardware connected to the host.

the only drawback is initial startup time (particularly pulling latest updates
after install) and archiving the vms (they're large, so fill up a laptop ssd).
i export to ovas on my backup system and then wipe the vm. another worry is
that virtualbox has been flakey recently
(<http://www.acooke.org/cute/VirtualBox1.html>
<http://www.acooke.org/cute/UbuntuonVi0.html>) - but ovas can be imported into
other hosts...

------
xentronium
For OS X tool see <http://boxen.github.com/>

~~~
druiid
Well, but the article was actually speaking out against Boxen a bit.
Essentially, you're still shoehorning all of the dev stuff onto your local
machine with Boxen. It's much better than installing it all manually, but
still not a separate dev environment.

------
pixelcort
How about the best of both worlds? A common VM image with CLI tools and dev
server inside the guest OS, with VirtualBox folder sharing to permit a GUI IDE
and the git repo to live on the host OS?

~~~
tjbiddle
This is what we do now - It gets to be a bit of a pain later. I'm looking
forward to switching to Vagrant in the future (On the backburner for now as I
have other things on my plate) - Vagrant allows you to easily nuke and re-
setup your entire environment, and change things easily for the next go around
- while if you're using a VM image you'd need to re-image every time you make
changes that you want to stay persistent.

------
vinitool76
How is this different from Boxen (<http://boxen.github.com/>)? You have
exactly copied the central idead around it and made a blog post.

~~~
hoov
The two ideas are very different, but complementary.

Boxen is about getting your MacOS environment set up with tooling.

The author of the original blog posting is talking about how to automate the
creation of a production-like environment in which to do your development.

The creation of that environment requires a few tools installed on your
physical Mac, which could be managed via Boxen. For example, you might use
Boxen to automate the installation/upgrade of Vagrant or your IDE of choice.

------
rbsn
I have used Vagrant in the past, and I had to stop using it. Shared folders
were just too slow. 2 seconds to refresh a page was too much when you're
trying to be productive.

~~~
hosh
Huh, that's interesting. How old is your hardware?

------
khitchdee
Points towards generative programming?

------
sandGorgon
has anybody tried Vagabond (<https://github.com/chrisroberts/vagabond>) to do
the same thing without a VM ?

------
EliRivers
No no no, I don't like automation; I like automating things :)

------
zerr
Couldn't this be shown without a blasphemy?

~~~
taybin
Where is the blasphemy? The terrible restoration of the fresco?

~~~
zerr
Shown in the context of this article.

~~~
taybin
I re-read the article and still don't know what you're talking about.

------
jpitz
Monoculture used to be a dirty word.

