
Show HN: Posixcube, a shell script automation framework alternative to Ansible - schizoidboy
https://github.com/myplaceonline/posixcube
======
bsuh
A similar tool I switched to from ansible for my personal project is
[https://pressly.github.io/sup](https://pressly.github.io/sup). Ansible felt
heavy and restrictive for my small project (lots of abstractions to learn and
APIs on top of that for extensibility if those abstractions don't meet your
needs).

sup on the other hand has a very small surface area of things needed to learn
if you're familiar with shell scripting. I would definitely recommend it for
smaller projects.

~~~
schizoidboy
I like a lot of things about this including, first of all, the website
describing it, and the screencast, and easy-to-follow instructions. It doesn't
meet my needs including handling of encrypted variable substitution in file
templates, but thanks for sharing.

------
boardwaalk
Hrm, personally I'd prefer a better language than Bash.

I was annoyed by all the structure Ansible, Chef, etc impose, so now I use
Fabric [1]. It's for a few personal servers, and for that it works just fine.

[1] [http://www.fabfile.org/](http://www.fabfile.org/)

~~~
notheguyouthink
I agree, bash is not that enjoyable. Not only that, but the standard binaries
in Unix tend to have non-consistent arguments and in general seem like a
clusterf--k to me. I know, it's my knowledge and my fault, but it doesn't mean
i have to think they're somehow elegant in UX, it's terrible, imo.

With that said, i _still_ prefer tools like Sup that just stick to bash &
unix. As much as i hate bash & unix, not using it requires learning yet
another API that probably will end up with either missing edge cases, or
complexity due to edge cases. Unix is the way it is because the "problem" is
complex, not because the devs involved were incompetent. An abstraction ontop
of Unix is just another API, for me at least.

------
jquast
I believe deployments should be described as declarative data, not as
imperative code.

Declarative: can first be tested, verified, dry-run, and implemented in any
language. salt (my preferred CM) has yaml data files that can be analyzed and
verified long before any system command occurs, for example.

Imperitive: You don't know what it does unless you run it. You don't even know
if it will execute without syntax error, especially in a dynamic language like
bash (posixcube) or ruby (chef, et al).

~~~
schizoidboy
Let's say I want to whitelist a set of IP addresses in the firewall
configuration. Let's assume I know the commands to do this using firewall-cmd.
Below is how I would do this with posixcube. Can you give an example with a
declarative system of your choice for comparison?

    
    
      # Loop through the list of servers passed in
      for cubevar_app_server in ${cubevar_app_servers}; do
        # Get the IP address of the server
        cubevar_app_server_ip="$(dig +short ${cubevar_app_server} || cube_check_return)" || cube_check_return
        
        # Skip ourselves by comparing to our hostname
        if [ "${cubevar_app_server}" != "$(cube_hostname)" ]; then
          if ! firewall-cmd -q --zone=trusted --query-source=${cubevar_app_server_ip} ; then
            cube_echo "Adding ${cubevar_app_server}'s IP ${cubevar_app_server_ip} to trusted whitelist"
            firewall-cmd --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return
            firewall-cmd --permanent --zone=trusted --add-source=${cubevar_app_server_ip} || cube_check_return
          fi
        fi
      done
    

This would be run as follows (where the -o parameter is expanded from
~/.ssh/known_hosts):

    
    
      posixcube.sh -h $TARGETSERVER -o cubevar_app_servers=*.mydomain.com -c firewall_whitelist

~~~
e12e
Generally the ip addresses would be selected by host groups (eg: all) - and if
"firewall-command" was the canonical way to add rules, that'd probably be what
eg Ansible would run on each platform.

An argument might be made that this would live better in a (templated or not)
config file in version control - and then the action become: fill in template,
push config, (re)load config.

In this case, for Ansible, you'd probably try to use command (there's also
she'll, but simple is better - when possible):
[http://docs.ansible.com/ansible/command_module.html](http://docs.ansible.com/ansible/command_module.html)

~~~
schizoidboy
I want to see a fully fleshed out example for comparison. When I find some
time, I plan to take this use case as an example, implement it in all major
automation frameworks, and publish the comparison.

~~~
e12e
Right, I suppose for Ansible the idiomatic approach is to write/use a plugin -
like:

[http://docs.ansible.com/ansible/firewalld_module.html](http://docs.ansible.com/ansible/firewalld_module.html)

Expanding the white-list of ips is a matter of config/templates.

Fwiw, I do think Ansible tend to get a bit complicated because plugins tend to
get a bit opaque and complicated. On the other hand, some simple special
variables like "all_other_servers" to avoid manually filtering out "self"
procedurally all the time is probably a good idea.

I'm not sure if I'll be able to spare the time - but I'll at least watch the
github repository - perhaps I can try and help flesh out some alternative
examples if you file an issue?

[ed: perhaps such things should go here?

[https://github.com/myplaceonline/posixcube/issues/4](https://github.com/myplaceonline/posixcube/issues/4)
]

~~~
schizoidboy
Agreed. I referenced this discussion in that issue and hopefully soon I'll get
around to building it (would be wonderful if you could help). Thanks for the
pointers; they'll save me some time learning Ansible.

------
schizoidboy
The impetus for this project was that I wanted to move away from Chef because
it was costing me an extra administrative server per month. I wanted to try
something new, so I didn't bother looking into chef-solo. I wanted something
agent-less, and it seemed like Ansible was the latest alternative, but when I
looked at its YAML, I felt fatigue at learning yet another domain specific
language. I thought, "Hey, why not shell scripts?" (insert jwz RegEx joke
here). I checked search engines and Stack Overflow, but I couldn't find much
except for FSS which didn't meet my requirements, and I had always wanted to
learn more about shell scripting, so I created my own framework called
posixcube.sh (open source, MIT license):
[https://github.com/myplaceonline/posixcube](https://github.com/myplaceonline/posixcube).
There were a few requirements that I had from my Chef cookbooks:

    
    
      * Idempotent file templating with variable substitution
      * Plain-text and encrypted variable files
      * Logical packaging of recipes
      * Roles and a way to check for roles in scripts
      * Repeatable sets of actions for certain types of servers
    

So beyond the basics in posixcube.sh of uploading itself to the remote host
and providing a consistent API, the above requirements were solved with:
cube_set_file_contents, gpg for encryption, "cubes" for logical packaging
(although loose scripts and straight commands are also supported), -r ROLE,
and cubespecs.ini, respectively.

After having migrated an entire Ruby on Rails architecture
([https://github.com/myplaceonline/myplaceonline_posixcubes](https://github.com/myplaceonline/myplaceonline_posixcubes))
to posixcube, I thought it went very well, although I certainly see some of
the downsides of shell scripting such as checking for error codes in a
pipeline, strange HEREDOC interpolation, etc. I also see some of the benefits
of complete idempotency, but I didn't find the additional procedural checks to
emulate idempotency (outside of file templates) that difficult. For the
simplest cube example from the previous link which configures an NFS server,
see
[https://github.com/myplaceonline/myplaceonline_posixcubes/bl...](https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/nfs_server/nfs_server.sh)
and for a more complex nginx+Rails cube, see
[https://github.com/myplaceonline/myplaceonline_posixcubes/bl...](https://github.com/myplaceonline/myplaceonline_posixcubes/blob/master/web/web.sh)

You can test it as simply as:

    
    
      git clone https://github.com/myplaceonline/posixcube
      cd posixcube
      ./posixcube.sh -h $SERVER "cube_echo Hello World"
    

For more complex examples, see the usage. I'm open to any feedback and any
philosophical debate on this approach versus Ansible, Docker, etc.

~~~
mwpmaybe
I can't criticize someone rolling their own foo for learning's sake (or just
for the fun of it), but I think if you'd given Ansible a try, you would have
been happy with it.

~~~
cpburns2009
Ansible is nice, but it has some annoying quirks:

\- YAML quoting.

\- Quasi Python expressions from Jinja 2.

\- Which fields are expressions and which ones are strings is not always
clear.

~~~
mwpmaybe
Agreed, although I will say the templating has improved over the past year to
be more consistent and full-featured cf. Jinja 2 proper, and they are now
enforcing interpolation instead of inferring expressions (e.g. if foo is an
array you have to write `with_items: "{{ foo }}"`; `with_items: foo` was
deprecated, and is an error as of Ansible 2.0 IIRC). The quoting issue is
still there, although once you learn the two or three rules it really isn't a
day-to-day issue, and Ansible tries to warn you if and when it sees something
problematic.

------
huherto
So what happened to Perl ? Back in the day we used to do a lot of the sysadmin
automation using Perl. It was pretty awesome. Do sysadmins still use it?

~~~
schizoidboy
Perl would play nicely with posixcube. Just do cube_package install perl, and
then do with it as you wish. Posixcube is more about handling the things
outside of Perl, Python, Go, etc. like distribution, installation, etc.

------
tmaly
thanks for sharing, I needed something with a bit more automation for my side
project. I used small scripts, but I never had a chance to automate more than
that.

~~~
schizoidboy
You're welcome! Hope you might find this useful, and glad to hear feedback.

