
DevOps from Scratch, Part 1: Vagrant and Ansible - Kaedon
https://www.kevinlondon.com/2016/09/19/devops-from-scratch-pt-1.html
======
matt4077
Ansible seems to be the best of the bunch, at least for my use-case which is
just a handful VMs (Chef seemed to require more infrastructure than what it
would be managing).

So let me rant about it for a bit:

\- Whatever that language is, it's not something invented on earth. I've given
up on learning when to use quotes or how it wants its indentations and just
copy&paste&destroy everything&restart when it doesn't work after 15 minutes.

\- There are a few thousands tests in their test suite, which makes it quite a
feat to introduce a bug that breaks what amounts to "passing variables to a
subroutine":
[https://github.com/ansible/ansible/issues/17356](https://github.com/ansible/ansible/issues/17356).
(2 hours today I won't get back). I seriously don't know any other software
where I spend most of my time reading through github issues.

\- It's a tool to manage hosts. Which would make me think that "reboot"
shouldn't be a dirty word. But there's actually no function to do (a)
```/bin/sh reboot```, (b) wait for it to come back. First request I found is
four years old:
[https://github.com/ansible/ansible/issues/513](https://github.com/ansible/ansible/issues/513).
The current request has a nice summary of the cottage industry of "blog
articles that explain how to reboot & wait":
[https://github.com/ansible/ansible/issues/16186](https://github.com/ansible/ansible/issues/16186)

\- Projects are structured in a quite "sparse" folder structure. A small
project usually has 25 files in 20 folders and 18 of them are "main.yml" which
is probably a conspiracy to annoy me. The main feature an editor plugin for
ansible needs is not syntax highlighting but "use second-to-last folder name
instead of filename on tabs".

-

~~~
rjprins
\- Ansible lacks proper error handling

\- YAML is not a proper programming language

\- It's not easy to create reusable "functions". You can include tasks but it
can be painful to manage variables, you can depend on roles but everything
ends up in separate folders and files.

In retrospect I would prefer to just use Fabric, or use Ansible modules
through Python.

~~~
vacri
YAML is a config language, not a programming language (Yet Another Markup
Language).

If you want to configure your hosts using a programming language, just use
shell scripts.

\--

edit: the acronym has been changed to 'YAML Ain't Markup Language', to
separate it from documentation languages

------
Annatar
How much more flapping there needs to be? Chef, Puppet, Ansible, Salt,
Vagrant, Docker, CoreOS, runC, Kubernetes, OpenStack, the list goes on and on
and on. People are going way, waaayyy out of their way just so they wouldn't
learn shell programmimg and how to build all this automation with OS packages.
Is learning N amounts of discrete orchestration really easier and simpler than
writing shell scripts and encapsulating them in OS packaging? Seriously?!?

Enough with the flapping and discrete solutions already!!!

~~~
mwpmaybe
I'm not sure what you mean by "flapping," but the technologies in your list do
not all address the same problem space. You might as well ask why we need both
Solaris and C++. Only the first four (Chef, Puppet, Ansible, and Salt) are
configuration management tools, which take the place of "writing shell scripts
and encapsulating them in OS packaging." If you're well-versed in one, there's
no need (short of a job change) to learn the others.

Vagrant allows you to create reproducible development environments _using
shell scripts_ (by default), so it should be right up your ally. Docker is
used to containerize applications. runC is the container runtime used by
Docker. Kubernetes orchestrates containerized applications across a fleet of
CoreOS (a minimal "Docker host" Linux) machines. OpenStack orchestrates your
data center. They are discrete solutions because they do different things.
Each piece of the puzzle does one thing and (hopefully) does it well. That
philosophy should sound familiar to you!

Back to "writing shell scripts and encapsulating them in OS packaging": have
you _tried_ a configuration management tool? I used to be old school like you;
specifically, maintaining a fleet of AIX instances using a healthy mix of
Perl, ksh, and RPM. I too was resistant to the tide of new, quickly-evolving
technologies. I started using Ansible about a year ago and it's changed my
life. Don't be so quick to pooh pooh the idea that there just might be a
better way to do things.

~~~
Annatar
_I 'm not sure what you mean by "flapping,"_

Flapping, you know, when a person who cannot swim is drowning and just
flailing and flapping their body, most notably arms, all over the place in an
attempt to stay above water so that they do not drown.

 _but the technologies in your list do not all address the same problem
space._

I don't know about you, but for me to get a fully configured, ready-to-serve
system, I need both provisioning and configuration management. With that being
the case, provisioning and configuration management fall under the same
problem domain.

 _Vagrant allows you to create reproducible development environments using
shell scripts (by default), so it should be right up your ally. Docker is used
to containerize applications. runC is the container runtime used by Docker.
Kubernetes orchestrates containerized applications across a fleet of CoreOS (a
minimal "Docker host" Linux) machines. OpenStack orchestrates your data
center. They are discrete solutions because they do different things. Each
piece of the puzzle does one thing and (hopefully) does it well. That
philosophy should sound familiar to you!_

I'm familiar with all of those technologies, and yes it does indeed sound
familiar to me!

But, what is unclear to me is, why I would use a discrete provisioning
solution like Kubernetes instead of using, say, imgadm(1M) and vmadm(1M)
through parallel SSH?

Why would I use a discrete, proprietary solution like Vagrant when I wrote my
own, simple variable expansion function which uses eval(1), and that's all
that's really needed for configuration management? I'm far, far better off
with a simple shell function then using Salt or Ansible. Simplicity, the core
of the UNIX principle. eval(1) is much simpler and easier to understand than
an entire provisioning and/or configuration management engine! Or do you
disagree?

Other than templated configuration excerpts and self-assembly, both easily
implemented with a few simple lines of shell code in a small shell library
which OS packages can source, what more do I need to configure systems? Why
make things unnecessarily complex? In fact, that exact same approach is what
Chef uses, conceptually, when one specifies the configuration management
policy, except that with my solution, the system administrator need not learn
a proprietary Ruby dialect and need not employ a client-server architecture;
they can just dive in and leverage their existing shell programming knowledge,
with a zero learning curve. And because it's all just shell and AWK, the
system administrator has infinite possibilities to configure the systems and
do anything they need to do, without being limited by the framework of
whichever technology they are using (like YAML or Chef recipes).

So why bother??? Why flap?

Why would I need runC and Docker to provision systems, if I can just install
OS packages which configure the system, and then make a ZFS image?

Or, in terms of Linux: why do I need say Docker and Kubernetes, if I have
everything in OS packages, and I just use KickStart to PXE boot, automatically
install and configure the target node with OS packages?

 _have you tried a configuration management tool?_

You bet your ass I did! But that's not the point, the point is, why should I
learn a discrete, proprietary systems, when I can just write a simple shell
library which leverages the eval(1) command to do variable expansion, and
BAM!!! I instantenously have a template expansion engine so all my
configuration, even code, can be parametrized using the same varaible
evaluation and expansion technology which the shell uses, and which the
configuration packages can call?

With that:

\- I don't have to use proprietary, discrete systems;

\- I can leverage technology already built into the operating system;

\- I don't need a client-server architecture like Chef or Puppet, ever (ssh-
sshd is just fine, thank you!)

Why would _anyone_ waste their time?

~~~
mwpmaybe
Your strategy of calling everything a provisioning system and then decrying
the surfeit and complexity of provisioning systems is deeply flawed.
Kubernetes is not a provisioning system. Docker and runC are not used to
provision systems. Vagrant is not "an entire provisioning and/or configuration
management engine," and its functionality cannot be trivially implemented with
a "simple variable expansion function." You say you are familiar with these
technologies, but your familiarity is insufficient. Your comparison of Docker
and Kubernetes to OS packages, Kickstart, and PXE is outright laughable. You
clearly do not understand the various purposes of these tools.

You point to imgadm and vmadm as superior solutions, but these appear to be
"proprietary" (to use your own word) to SmartOS. Why not add SmartOS to your
list of unnecessarily technologies and apply the same critical thinking?
Triton appears to occupy the same space as OpenStack (Triton DataCenter),
Kubernetes (Triton ContainerPilot), CoreOS (Triton SmartOS), and possibly
other components, and it appears to use the same kernel-level containerization
facilities as Docker and runC. You're using a competitor's vertically-
integrated solution, so no, you don't need another stack. If you weren't using
Triton you would be researching these alternatives. Your argument is akin to
saying you don't need a car because you have a Buick, people who drive cars
are using an unnecessary, proprietary mode of transport, and they should all
just drive Buicks instead.

That brings us back to configuration management tools. You keep calling them
proprietary, but the cores of all these products are open source! And while I
agree with you that the client/server model used by Puppet, Chef, and
SaltStack is unnecessary—although it does have benefits at scale—it's worth
noting that Chef Zero and Ansible _just use ssh-sshd_. These tools are making
the same system calls as your shell scripts, they're just doing them from Ruby
or Python instead. So just to recap:

 _> I don't have to use proprietary, discrete systems;_

They're not proprietary, they're open source, and you only need one of the
four.

 _> I can leverage technology already built into the operating system;_

They do: Ruby or Python and system calls.

 _> I don't need a client-server architecture like Chef or Puppet, ever (ssh-
sshd is just fine, thank you!)_

Okay, so use Chef Zero or Ansible.

Finally, your other main argument, against abstraction, does not hold water.
Everything we do is an abstraction. Shell scripts are an abstraction. Why
aren't you writing your configuration management in C? (Wait, why are you
writing it in C instead of assembler?) On one hand, you have a portable, open
source tool with a domain-specific language. On the other hand, you have each
company creating their own proprietary (this is the correct use of
"proprietary," by the way) tooling, and an engineer has to learn it from
scratch every time they change jobs. You may think any engineer can walk in
off the street and instantly grok your homegrown shell- and AWK-based big ball
of mud, but you are probably wrong. If your solution is as good as you think
it is—if it's portable, extensible, and easy to maintain—you should release it
and we can add it to the list of available CM tools. If it's not, why are you
reinventing the wheel?

~~~
Annatar
_Kubernetes is not a provisioning system._

Really? What is it then? Let's see...

 _Kubernetes (commonly referred to as "k8s") is an open source container
cluster manager originally designed by Google and donated to the Cloud Native
Computing Foundation. It aims to provide a "platform for automating
deployment, scaling, and operations of application containers across clusters
of hosts"_[1]

Automating deployment? I guess _deployment_ has nothing to do with
_provisioning_ a resource... really, if you are going to argue for these
tools, you should understand the concepts which they are trying to solve. Or
are you one of the developers of one or some of these tools, and this is just
rubbing you the wrong way, because it means that your tool isn't required? If
so, just state that upfront, so everybody knows where you stand, and we don't
have to discuss this any further.

[1]
[http://kubernetes.io/docs/whatisk8s/](http://kubernetes.io/docs/whatisk8s/)

Moving right along...

 _your comparison of Docker and Kubernetes to OS packages, Kickstart, and PXE
is outright laughable. You clearly do not understand the various purposes of
these tools._

Okay, then you explain to me _why_ I need Docker to do what I can do with PXE,
KickStart, and OS packaging? This ought to be really good...

 _Why not add SmartOS to your list of unnecessarily technologies and apply the
same critical thinking?_

Because SmartOS uses zones, which give one fully isolated, true UNIX servers
running at the speed of bare metal, and because it uses ZFS, fault management
architecture (fmadm(1M), svcs(1), and svcadm(1M)), performance and post mortem
analysis tools, and because it has everything required for massive datacenter
and cloud deployments, built right into the operating system from the
beginning, and not as an afterthought, knee-jerk "oh shit, we have to scale
Linux to massive deployments now" reaction. By professional system and kernel
engineers with formal education in those problem domains. imgadm(1M) and
vmadm(1M) in SmartOS are the functional equivalents of JumpStart (or
KickStart, in redhat's jargon, and AutoYaST in SLES one). That's _why_.

 _They 're not proprietary, they're open source, and you only need one of the
four._

 _Proprietary_ means that someone owns it, or that it is _specific_ to
something or someone, _not_ that it is closed source.

A product can be both proprietary and open source, and _freeware_ all at the
same time if the _proprietor_ is providing the source code and giving the
product away for free.

 _They do: Ruby or Python and system calls._

Neither of those _programming languages_ are part of the operating system
_consolidation_ on a proper UNIX, and on Linux, they are provided in various
versions by various people who are volunteers. And anything which is not part
of the _OS /Networking consolidation_ on a _real_ UNIX is a third party and
unbundled _application_. That's the only distinction we have ever needed, and
the only distinction we have ever made.

This mistake was already made once on UNIX, when Perl was hip just like Ruby
and Python are now: we are still coping with the fallout of ripping out, and
replacing all the Perl garbage which has since been integrated into the
OS/Networking consolidation. You haven't yet been busted by that on Linux, but
you will be, because Linux keeps repeating the same mistakes which UNIX made.
Except we learned from that, and on Linux it just causes more flapping.

Between Ruby, Python and shell and AWK, I will always pick shell and AWK (and
not even bash, but traditional Bourne shell), because they are stable, the
interfaces won't change, the syntax will stay the same forever, they are well
understood, easy to learn, and they are small and fast, very fast.

I have regularly beaten Python with AWK both in code density and speed
(latest: 487 lines of Python code in 183 lines of AWK, and no kernel hooks
required like with the Python code). So if you're implying that Ruby and
Python are ideal for massive deployments and configuration management just
because they're _hip_ right now, that's insane, and I'm not buying it.

 _Okay, so use Chef Zero or Ansible._

Why? What does the Ruby dialect of Chef offer me, that I couldn't do in %pre,
%post, %preun and %postun in a RPM package? Policy? Policy is best solved with
a throughly thought through _change management process_. Capability Maturity
Model and all that, you know?

 _Why aren 't you writing your configuration management in C?_

And just what do you think svcadm(1M) is written in?

 _(Wait, why are you writing it in C instead of assembler?)_

Who says I'm not? Use the right tool for the job. The problem here is, none of
these Chefs, Puppets, Ansibles, Salts, Dockers or Kubernetes are the right
tools for the job. Even Linux as the substrate for all of this is not the
right tool for the job! But then again, judging by the other comments here,
people are starting to realize that, and that's a good thing.

------
lazyant
Just a note that Upstart is Ubuntu-only and it's actually going to be replaced
by systemd.

~~~
mwpmaybe
It already has been replaced by systemd as of 15.04 (and 16.04 LTS).

------
sp527
I like the premise of this, but can someone link a more modern guide that
deals with Docker, ECS, etc?

~~~
Kaedon
Looks like this is a little closer to what you're looking for I think:
[https://medium.com/aws-activate-startup-blog/running-
service...](https://medium.com/aws-activate-startup-blog/running-services-
using-docker-and-amazon-ec2-container-service-bde16b099cb9#.fhj8j5rj9)

~~~
sp527
Thanks. For anyone interested: the above linked medium post doesn't mention
config management, but I see that Ansible has some form of ECS support.

------
hayd
Why go through all this for a simple flask app (or even a more complicated
one)? What do you gain over just using Elastic Beanstalk (which handles the
monitoring/autoscaling etc.)

~~~
aisofteng
I feel it's pretty clear that the purpose of this article is to illustrate
this general methodology using these tools and is not "how to deploy a Flask
app." The nature of the app itself is largely inconsequential to what is being
illustrated.

~~~
hayd
I guess my question was "why use vagrant/ansible over a more simple AWS setup
like Elastic Beanstalk"?

I agree it being Flask is largely irrelevant.

