
OpsMop – Config management and app deployment from the creator of Ansible - yankcrime
https://github.com/opsmop/opsmop
======
syn0byte
DSLs always seem like terrible relationships. Fun and exciting at first but
soon she's telling you what to wear, what to say, who you can hang out with,
etc. Oh sure, you can "Do what you want", and "It's fine." But you know she
will punish you for it later when you least expect it. You would break it off
but now you have so much invested she is integrated into your life. Maybe she
will mellow out if you "put a ring on it" and buy a support package...

~~~
tyingq
I may be wrong since I just skimmed the docs, but this looks like an attempt
to allow general purpose python when needed, rather than being fully locked
into a DSL.

~~~
mpdehaan2
Righto (author here)! It's a lot more relaxed (IMHO) than some. Most methods
are raw python. The "set_foo" methods are just methods that get called when
the objects are constructed to populate certain fields. You can just pass
those in and leave them off if you want.

------
sciurus
The question I don't see answered anywhere on their website or repo is "why?".
Why in 2018 with

a) many mature configuration management systems like puppet, chef, ansible,
and salt

b) lots of architectures that no longer require traditional configuration
management (e.g. "serverless", immutable deployments of VM images, docker
images on hosted kubernetes)

is there a need for a new CM tool? What is OpsMop doing different or better
that gives it a hope of gaining traction?

~~~
ElijahLynn
Michael DeHaan, creator of Ansible, wrote an intro here:

OpsMop: Building the Next Generation of Configuration Management Tooling ==>
[https://medium.com/@michaeldehaan/opsmop-building-the-
next-g...](https://medium.com/@michaeldehaan/opsmop-building-the-next-
generation-of-configuration-management-tooling-11268b7f21b6)

~~~
ElijahLynn
Asked Michael DeHaan on Twitter today about the "Why?"
[https://twitter.com/ElijahLynn/status/1075446656533811200](https://twitter.com/ElijahLynn/status/1075446656533811200):

"Efficiency in both operation (speed) and development of plugins is a big
one."

------
honkycat
As a "devops engineer" _cough_ sysadmin _cough_ , configuration management is
still a freaking nightmare.

I write too much YAML, I do not know why everyone all at once decided they
were done building abstractions and instead wanted to write a ton of
boilerplate.

I use ansible currently for our SMALL business. I have thousands of lines of
ansible to do relatively simple tasks. YAML is not a programming language, you
have to repeat your self over and over again. If you get tired of doing that,
you are out of luck. You are not writing in a programming language so you
cannot develop an abstraction.

That being said, the main advantage of Ansible is that someone has already
gone in and written all of the boilerplate code for provisioning various
resources in my cloud provider. Unless OpsMop has this, I don't see the value-
add.

~~~
jacobsenscott
I switched from chef to ansible because in my opinion a DSL is the wrong way
to go for this stuff. Granted I've only ever managed fairly small deployments.
But so much more can go wrong when you are writing code vs writing yaml. If
your yaml parses it will probably work. I write ruby application code everyday
and I love ruby. But I hated writing chef scripts. I also ended up with many
fewer lines of yaml than ruby.

~~~
scirocco
Requiring you to be a developer to do automation is not a sustainable
strategy. Ansible is a better fit for non-developers to do automation, which
helps drive adoption

And since it's agentless you can automate almost anything... As mentioned
above - the playbooks created by the community avoids you from reinventing the
wheel. Companies like Cisco are writing playbook to automate their switches

------
INTPenis
This is ansible for people who like Python.

It's kind of sad because I feel I've invested 5 years into Ansible but now
Michael releases something equally impressive seemingly based on experiences
and lessons learned while building ansible.

So it's very hard not to just abandon Ansible and go with this new shiny tool.

~~~
jvehent
They are just tools. People will come up with better ones every other year,
and we'll abandon the old stuff. That's ok, components should be replaceable
without having to reinvent the whole thing.

~~~
dvtrn
_That 's ok, components should be replaceable without having to reinvent the
whole thing._

A fine, admirable and accurate (IMO) mindset. Things get weird when recruiters
find out about these new tools and suddenly we're having that time honored
discussion about how 'tech hiring is broken' because the people screening
resumes don't realize OpsMop hasn't been out for 4 years and no one has 4
years experience with OpsMop.

Everyone scrambles to learn OpsMops-so they can get a job where OpsMop ends up
being abandoned and the DevOps team goes back to ansible, other teams start
adding it to their tool chains because someone got hired who fiddled with it
once and becomes an OpsMop house, a few engineers get very good at it, realize
they can probably write a better version, three years later someone's posted
"Show HN: OpsBroom" to the front page.

Repeat.

------
VectorLock
Although I mostly loathe DSLs and have been wanting something almost precisely
like this for quite some time looking at the examples I think I might prefer
the Ansible YAML approach better.

Ansible YAML is definitely more tidy and concise despite Python's awesome
readability. And even though you get all the benefits of the full Python
language is a definite advantage (try turning a list ["www1", "www2", "www3"]
into the string "www1:443,www2:443,www3:443" without plugins or some serious
ugliness) I wonder if having Ansible/Jinja2 preclude most of those abilities
is a feature and not a disadvantage.

Either way as someone who has just found himself in an environment that uses
Chef/Ruby for Ops stuff and Python for everything else I'll be watching this
closely.

~~~
rmetzler
I agree that your problem is much harder than it should be and I think it is
due to the format() filter not being fit for use in map(). Nevertheless here
are 3 versions how you could solve this problem in Jinja2:

    
    
        {{ list | map('regex_replace', '(.*)', '\\1:443') | join(',') }}
        {{ list | zip_longest([], fillvalue='443') | map('join', ':') | join(',') }}
        {% for item in list %}{{item}}:443{%- if not loop.last %},{%- endif %}{%- endfor %}
    
    

I wish the following would be possible:

    
    
        {{ list | map('format', '%s:443') | join(',') }}

~~~
mpdehaan2
My historical argument was always that if a playbook looks ugly, it is wrong.

Over the years I haven't been around to keep saying that, but I'd be inclined
to have to find ways to open up the programming model while still keeping the
mostly organized guide-rails in place.

------
athenot
Out of curiosity I checked it out. This feels a lot like Chef or Puppet DSLs.
On the pro side, this gives you flexibility to write whatever convoluted logic
you want in code. But that also means you need to know the quirks of each DSL
once you start to deal with modules.

And despite them calling the language a DSL, it really is a subset of a
general purpose programming language, so there's if readability is a goal,
well there's going to be an unavoidable amount of line noise.

As a matter of personal taste, I find this to be less readable than Ansible's
YAML. Though it's not the most elegant way to express things, it's still a lot
cleaner and predictable than code.

In all fairness, the author addresses this in his documentation:

> You might like the language of one thing more than the other. Pick that
> thing.

------
drcongo
This looks promising. We still use ansible because we have a lot of time
invested in the roles we've built, but I _so_ hate the yaml DSL of it and all
I ever want when writing a new role is the ability to debug it properly with a
decent toolset and logging.

------
ensignavenger
I was worried this would be proprietary commons clause licensed like the
creators vespen project... but I was pleasantly surprised that is is open
source, Apache (real Apache) licensed! Further, it appears that Vespene has
been open sourced under the Apache license as well!

Thank you Michael for creating these projects and releasing them as Open
Source! I am a big user of Ansible, and have often wished there was a good way
to write plays in Python. I will be keeping a close on on both of these
projects!

~~~
mpdehaan2
You are welcome! Yeah the thing I was worried about with that kinda went away.

Feedback would be welcome, stop by the forum sometime!

------
aasasd
So. With Ansible's YAML it was always a pain when your configuration grew
beyond what's envisioned and you needed some more conditions and loops―because
the programming functionality was always incomplete and needed to be
implemented in every new case (e.g. you couldn't do loops of roles when I last
checked).

I guess Mr DeHaan decided that this was a mistake and returned to the beaten
path of full-featured programming. In which, of course, you can't spill config
_data_ into your files as freely as you would with YAML, since it has to be
distinguished from language constructs and thus encrusted in syntactical
markings.

Now, it seems to me that Lisp better marries data structures to occasional
programming functionality, especially for Ansible where every part of a config
is a declarative value first, and an instruction secondly. More so compared to
Python, which uses statements instead of expressions in plenty of notorious
cases.

Of course, you'd still have to mark values with more syntax than in YAML's
free-flow, and I guess Lisp isn't good at embedding expressions in strings. In
this regard, I'm very curious to see one day a lightweight version of Lisp
syntax reminiscent of YAML. Notably, Ansible's configs essentially use the
syntax of "function: argument -keyword: value -keyword: value..."

I'm not sure if I've seen a Lisp-based Ansible clone. There's Vektra's Tachyon
in Go, but apparently it only uses Lisp for embedded condition snippets:
[https://github.com/vektra/tachyon](https://github.com/vektra/tachyon)

~~~
mpdehaan2
"I guess Mr DeHaan decided that this was a mistake and returned to the beaten
path of full-featured programming."

Nope, and I've written on this a bit -- I think Python is a better fit for
those that want to program in Python.

For those that want to build their own CMS in something else, go for it!

------
peterwwillis
If anyone modifies this thing to use real YAML (not whatever Ansible was
passing off as YAML) or HCL as the configuration file format, please post it
to HN? I'm never going to use a programming language DSL.

What I really want is a real document standard that describes common
configuration management functions, and then implement a parser in every CM
tool, so all of them can do the same basic functions with one standard file.
Like a k8s file that can describe Terraform-like modules/providers/resources,
and each CM can have it's own implementation to build, test, and apply them.

As an aside, we really aren't using YAML to its full potential. We could
benefit from an implementation of YPath so we can work with YAML files the way
we use XPath with XML, and tags are greatly underutilized.

~~~
rkangel
Whereas I am _only_ willing to use a programming language DSL. In any
interesting real world case, you are going to run into cases requiring some
form of arbitrary logic. Some DSLs will provide you functionality to do loops,
ifs etc., but you can't beat a real, actual, programming language.

~~~
peterwwillis
That's assuming you should be adding logic. The whole idea of CM is to get
away from writing programs. Logic adds complexity, is subject to bugs, and is
hard to maintain. Most logic I've seen is merely adding procedural code rather
than declarative, which is a terrible pattern for a CM.

I have worked with nightmarishly complicated Ansible playbooks and roles
designed to provide declarative infrastructure when it didn't exist in a
module. If all you had to change was a group_vars setting, you could be
confident in your change. But if you had to edit a role, playbook or task,
nobody was sure if it would break something in the future or not. It was much
less robust than a real module because it was just the DSL.

With a Python DSL you could write code that was declarative, but because the
language is complex, lexing is more difficult, and simple changes to
configuration become more error-prone.

By moving to images and containers, the majority of configuration management
is now largely unnecessary. We will always need a CM tool, but they should
give users _less_ rope to hang themselves with, not more.

~~~
mpdehaan2
First off, it sounds like your playbooks sucked - sorry about that, with great
power comes great responsibility. Even as I wanted to make things simple, all
things could be abused, and in adding more flexibility I've seen some evil
things done in my name :)

So as immutable systems grow - that's a damn good thing. I personally think
the docker ecosystem is a trainwreck currently, but that's only _my_ opinion,
and doesn't really affect anyone else. But the same concepts have been around
for ages in EC2 in immutable systems.

What typically happens is people use the CM tool to provide a higher level
abstraction than Bash for describing the image build, so they get features
that should have always been in docker files but never were (templates are the
biggest one for me).

Further, there is always a need to deploy the "undercloud". Even in heavily
containerized microservices setups, there's a need to handle all the magic
upgrade fun of upgrading a Mongo cluster, a Cassandra cluster, or a Kafka
setup.

Whether we call it configuration management, app deployment, or just a
structured way to push scripts around and see what failed, we're going to need
those tools.

I believe immutable systems is a fantastic philosophy where it works. It
doesn't work _everywhere_. And CM tools still have a niche for those that want
to use them in the build process.

If folks want straight docker files instead, I'm also cool with that.

------
thinkingkong
This is an early stage in the project but it looks like its intending to be a
Chef or Puppet in python with a terraform-esque plan and apply stage.

~~~
IfOnlyYouKnew
Isn’t Ansible a “Chef or Puppet in Python“?

~~~
dsmithatx
Ansible is an imperative language while Puppet is declarative.

Ansible is push mode (ssh connection) while Puppet and Chef have an agent that
pulls and enforces configuration.

Ansible is great for quick hacky or one time solutions like building a server.
It's very newb friendly and easy to use out of the box. Puppet is great for
enforcing a configuration every 30 minutes and scales across Data Centers. In
my opinion Chef is more robust and is easier to use and share cookbooks on
Supermarket.

Ansible Galaxy isn't as great for sharing work, but I find Ansible is so easy
to use I don't need anyone else's work.

Once this has a pull mode (agent) it seems it will be like a very lightweight
Puppet like declarative tool. Both Puppet and Chef have very mature products
for serving and managing infrastructure. They can scale.

This tool might be awesome years from now, but I have to use Ansible everyday
at work. From 2.0 to 2.8 it has become a solid and fairly mature product. I
also use Puppet and have for 10 years. It feels old and is a pain to use
across 1000s of machines in an Enterprise. There is a learning curve and
training to be done at all times. At my last job I used Chef and I really love
it for being powerful, easy to use, and just having a great design.

We currently install Puppet with Ansible once initial config is done to
enforce policies. I could see this potentially replacing the Puppet side of
that equation some day.

I'll for sure watch this tool and experiment with it in my lab as it grows. I
do love Python and Puppet is far to long an overhaul. It will be interesting
to see what directions are taken as this new product matures.

I do not think this is about using python instead of yaml. It is super easy to
write Python modules for Ansible and use them. We have to use them for the
things that aren't included in the base such as managing Infoblox etc.

~~~
jabl
There is ansible-pull. It's not as useful as one might initially think, as
there's lots of ansible code written assuming push mode.

------
jarvuschris
Check out habitat if you like the sound of this:
[https://www.habitat.sh/](https://www.habitat.sh/)

~~~
mpdehaan2
I'm a big fan of the Chef people and they are good folks. Habitat is about a
way to distribute applications to remote nodes and is somewhat an alternative
to docker. It's kind of apples and carbeutors. Just like Habitat does not
compete with Chef, Habitat and OpsMop are also very different things.

Chef is a closer thing to compare it to.

------
StreetChief
I just want to say the name OpsMop is awesome.

------
thesorrow
I may be late to realize this but Python is becoming a mandatory language to
master for DevOps people.

I guess it's time to learn ...

------
je42
uhm. no unittests for the project ? :((

~~~
mpdehaan2
Valid point. I started this about a month ago, and as APIs were evolving I did
not want to slow my velocity by having the tests fight against shifting it
around.

I suspect the first release will be in about Feburary, but the time to
collaborate on working on it is now.

It is designed to be fairly testable, but if you've ever written a similar
system, you know that distributed testing of modules on a matrix of 12
platforms _IS_ a trainwreck.

Any units would mostly be proving out the language.

The best way to do that is with a specialized callback, that shares the event
stack, and we can then introspect that the events happened in the correct
order.

As the callbacks have been changing too much now, writing a boatloat of tests
would have been a mistake.

At this point, I'm most interested in feedback on language features, and these
will come.

It was an absolute mistake that ansible itself was not more testable at a mock
level, a problem made a lot better now by everything being done in one
process.

Ansible had the ability to write modules in any language, but this was never
really used in practice.

------
jonathanoliver
I'm all for choice in configuration management. That said, I find this quote
on their page amusing:

> The purpose of the Opsmop community is to come together to build the
> ultimate configuration tool we all want to use.

Kinda reminds me of XCKD: [https://xkcd.com/927/](https://xkcd.com/927/)

~~~
mpdehaan2
I'm all for more standards! LOL.

Actually this was meant to describe what we do when we work on OpsMop. It's
like "hey, let's all figure out what we want in an ideal universe and build it
together". Versus like "hey, you're all my user peeps"

------
Walkman
It's interesting, we have developed a very similar framework internally for
our appliance and use it in our product for 4 years. We "present" files from
templates filled with variables coming from the user interface and those
presenters call "notifiers" (e.g. restarting a systemd service when a file
changed on disk).

