
Marelle: logic programming for devops - lars512
http://quietlyamused.org/blog/2013/11/09/marelle-for-devops/
======
calpaterson
I'm skeptical, because there is already a configuration management language
that is strongly influenced by Prolog and logic programming - Puppet - and the
results are not that great.

The secret of configuration management is that ordering NEEDS to be a first
class feature and ordered manifests should be the default, not the option.
Puppet suffers hugely from the influence from the fact that ordering is
unpredictable and that you have to use arrows or requires to add order. People
accidentally forget to add ordering, the resulting code works most of the
time, and then one day you start running into an issue because some implicit
dependency either didn't happen, or happened in the wrong order. In as much as
all logic programming involves not explicitly controlling flow, I think
they're inappropriate for configuration management. Sadly, I'm still stuck
with using Puppet because of the organisation I work in.

Having worked with Prolog in the past, I would also suggest that an important
feature of a configuration management language is that the syntax make it hard
to make typos and other minor errors because the feedback loop of deploying
your code against a local vagrant machine is already long enough. I vividly
remember the annoyance of having missed the full stop to end a block. I'm not
desperate to repeat that experience.

~~~
NuclearTide
I second everything said here.

In my internship last summer, I arbitrarily chose Puppet to automate the
configuration of the company's developer environments. Puppet's unpredictable
ordering should be a write-and-forget benefit, but in reality it's very
difficult to reason about. In the end, my script imposed a top-to-bottom
ordering with requires.

Also, developing configuration scripts against Vagrant is a huge pain. While
`puppet parser validate` checks for syntax errors, most of the logical,
dependency-based errors reveal themselves after Vagrant takes a good minute to
fire up.

Anyway, hopefully my Puppet-Vagrant scripts aren't causing any maintenance
nightmares...

~~~
lobster_johnson
Puppet has implicit ordering (for example, a file will depend on its parent
directory) as well as explicit (arrow syntax, new in 3.0, I think).

But really, I don't understand the complaint. Writing explicit dependencies
literally what Puppet -- or any other config management system -- is about.
Puppet cannot possibly know what your intent is. So you have to tell it. Just
like you have to tell the OP's system. It's simple enough.

Having used Puppet for a few years, my biggest complaint is that it's not
quite modular enough. For example, in one system I have a way of declaring an
"app". An app may have some dependencies (Postgres, Memcached), settings,
users, etc. I have two choices, I think; either have one large, monolithic
define allowing me to instantiate the app from a bunch of config:

    
    
        apps::app {
          "someapp":
            environment => 'production',
            uses_unicorn => true,
            db_name => 'someapp_production',
            db_user => 'someapp',
            memcached_servers => [...],
            # etc.
    

...which means that every setting has to be a variable that's passed around
and used by the define. Or I can split it up into pieces:

    
    
        apps::app {
          "someapp":
            environment => 'production';
        }
        apps::memcached_config {
          "someapp":
            servers => [...];
        }
        apps::db_config {
          "someapp":
            host => ...,
            db_name => ...,
            db_user => ...;
        }
        apps::unicorn_config {
          "someapp":
            ;
        }
    

The problem with the latter approach is that eventually, much the
configuration needs to be collected into just a few actual config files. Let's
imagine that the app uses a single "app.conf". Normally you would build it
using a template. But with the second approach, you would have to build it out
of fragments that come from multiple defines; probably you would collect them
in a conf.d type folder, and then use an "exec" definition to concatenate them
into "app.conf".

Puppet is very modular, but it's a bit awkward when it comes to this level of
modularity -- or I haven't found a better way.

~~~
calpaterson
> Writing explicit dependencies literally what Puppet -- or any other config
> management system -- is about. Puppet cannot possibly know what your intent
> is. So you have to tell it.

Other tools just execute manifests top to bottom, which is a much less error-
prone approach.

I agree that finding the right abstraction level for modules (and for reuse)
is not well solved.

------
acjohnson55
It's been a while since I've studied Prolog, but it's good to see it getting
some press. I learned Prolog before ever dabbling in Scheme or ML, and it was
just as much of a paradigm shift from imperative style. It's kind of like FP-
style pattern matching, but taken to a whole new level, such that the pattern
matching itself is pretty much the entire computational engine.

Rather than using functions, which take a number of arguments and return a
result, you use predicates on a number arguments. You build your program out
of clauses, which are conjunctions of predicates. A process called unification
[1] finds the argument values (if any) that satisfy an entire clause. By
sharing constants and variables between predicates, the valid results that
unification can produce are restricted to what the logic of your program
allows.

So your algorithms end up being, in many ways, description of what a valid
results look like, and the Prolog engine goes out and finds them. This
paradigm clearly maps nicely to domains with lots of rules, like decision
engines, parsers, and such. Sounds awesome, right? One caveat is that, in
practice, you as a programmer have to have a pretty good understanding of how
unification works in order to write performant algorithms.

This is all from several years-old recollections and a little bit of Wikipedia
to refresh my memory, so someone please chime in if I'm incorrect someplace or
leaving out key concepts.

[1]
[http://en.wikipedia.org/wiki/Unification_%28computer_science...](http://en.wikipedia.org/wiki/Unification_%28computer_science%29#Application:_Unification_in_logic_programming)

------
mickeyp
Prolog's a surprisingly good fit for many every-day tasks where people
otherwise invent DSLs or poorly-defined state machines and pattern matching
algorithms.

So how come we don't use it more often? Well, the reason is actually quite
simple:

Prolog's incredibly difficult to reason about once you grow past trivial facts
and clauses like in the OP's article -- the examples given by him would've
been introduced to you in the first lesson on Prolog in, say, University. The
complexities of green/red cuts, backtracking and the procedural nature of
Prolog makes it a really difficult language to truly learn -- much less one to
specify complex programs in.

I've never programmed in a language where 5-6 lines of errant Prolog code
could stump myself, a post grad and a professor before until I did Prolog.

~~~
keenerd
It is a shame really. Declarative programming is so much simpler to think
about and extremely powerful. I've been working on trying to make declarative
programming more accessible and while I personally have seen 60x improvements
in ability, I have been unable to convince any of my teammates to give it a
whirl.

~~~
X4
@keenerd try to build a DSL ontop of prolog and they'll come & play.

~~~
keenerd
It is a library that works with any language, no need to learn a new one. Two
people can even work on the same problem in different languages and seamlessly
combine their work into one result.

~~~
X4
interesting! what is it for?

------
e12e
Also like the bootstrap.sh - very nicely structured and easier to verify than
many others I've seen. I'd still prefer these scripts to output missing
dependencies that calling sudo -- but otherwise very nice.

That is, rather than:

    
    
          function install_git() {
              echo 'Trying to install git'
              case $(uname -s) in
                #...
                Linux)
                  if has_exec apt-get; then apt_update
                    sudo apt-get install -y git
                  #...
                  else bail "Unknown linux variant"
                  fi
                  ;;
                *)
                  bail "Unknown operating system $(uname -s)"
                  ;;
              esac
          }
    

Simply aggreagate a list of missing packages, and ouput (based on a similar
case-statement for detetining host):

    
    
        echo Please install missing packages:
        echo sudo apt-get install ${missing-deps}
    
    

[https://github.com/larsyencken/marelle/blob/master/bootstrap...](https://github.com/larsyencken/marelle/blob/master/bootstrap.sh)

~~~
lars512
Thanks! :)

It's a little aggressive in calling sudo mainly because I'm using it to
bootstrap new systems non-interactively. I don't want it to ask any questions
at all, just to go.

------
CraigJPerry
Prolog does seem a reasonable fit for the problem space.

That said, even CFEngine's long winded syntax (born of its commitment to
"promise theory") isn't so bad. In my experience I don't feel expressiveness
of the available languages is the problem most I need of solving.

The real problems not being tackled by the existing configuration management
tools are testing bundles/(play|cook)books/modules, cleanly injecting metadata
to facilitate reuse and the bundle debugging workflow.

~~~
atsaloli
What kind of metadata would facilitate reuse?

Have you seen the latest verbose mode in CFEngine (as of 3.5, June 2013)? It's
more compact to facilitate debugging.

How else could debugging workflow be improved?

~~~
CraigJPerry
Say I have a simple bundle for inserting lines in /etc/security/limits.conf,
it'd be good to have the bundle agent / edit_line which describes how to make
the change, completely separate from the data that describes user1 gets a hard
nofile of 64k.

There's lots of ways to achieve that from just doing selective usebundles from
various bundle files containing only vars to more flexible approaches; In our
deployment we've provided this as part of our bundle framework (we inject the
parameter data via modules protocol - so a box in location A applies different
data than a box under line of business B), they both reuse the exact same
(comprehensivly tested) code.

This means no code release for a data change. No need to train everyone in CFE
syntax just to allow an app developer to express that user c on box d is
allowed more open file handles.

The point though is that It'd be good to have as a first class language
feature instead of each cfe site reinventing a separate framework for the same
functionality.

Haven't had any hands on with 3.5 so not familiar with that, but better print
debugging isnt where I was going with the idea. I want to be able to step
through a bundle, I want to be able to break when global class X is set. I
want to then be able to set class X while stepping through. Kinda like making
debug logging level interative and with higher signal to noise ratio on
screen.

If there was language support for unit testing my bundles I could use that to
reason about what I expect to happen. Diego Zambonis book proposed one way to
do that with the existing facilities but its just a proof of concept, e.g. it
cant guarantee environment cleanup after a show stopping error in a test.

------
zimbatm
The meet/met notation instantly made me think of babushka:
[http://babushka.me/](http://babushka.me/)

The base principle seems to be the same except that babushka is a ruby DSL and
that the dependency solving is part of the library instead of the language.

~~~
qznc
Babushka seems to be one of the inspirations. The article says "Still,
Babushka made me uneasy: all this ceremony and complex templating, just to
describe a few facts and simple rules?"

------
e12e
Very interesting. I wonder if it would be viable to build an entire package
management system (like apt) in a similar manner -- and if there are any
(semi-recent) improvements to prolog-like systems that might improve
dependency resolution "for free"?

~~~
X4
These are the package management systems that I use. A new package management
systems just does this [http://xkcd.com/927/](http://xkcd.com/927/) What we
need is a cross-platform cross-applicaton meta package management system which
comes as configuration management that evolves to a general purpose package
manager without relying on any external package manager.

I'm not fanboy enough and would happily replace:

emerge, equo, apt-get, aptitude, ipkg, dpkg, zypp rpm, yum, tazpkg, pacman,
homebrew

tar/gzip, slackpkg (Yeah, you, Slackware, we were friends and you helped me
for long enough. You shouldn't do all my work anymore, you're old. Take a
rest)

pip, cabal, bundler, gem, composer, pear, cpan, ctan, cran, maven, vundle –
haven't you also a love hate relationship with these?

