
Show HN: Welder – set up your Linux server with plain shell scripts - pchm
https://github.com/pch/welder
======
StanAngeloff
I used to bootstrap a lot of VMs in development using shell scripts [1]. At a
certain point Bash scripts become unmanageable. Ansible, Chef, Puppet have
stricter syntax, battle-tested modules, large communities, proper multi-OS
support, etc. Investing time to learn any of those tools certainly pays off in
the long run.

Bash is great for a quick setup or a one-off instance. However it requires
discipline for anything more. You are also dealing with the underlying OS
yourself (e.g., Nginx is packaged and configured differently in Ubuntu,
Debian, Cent OS & Arch; networking is configured differently in Ubuntu
depending on whether systemd is installed, etc.).

[1] [https://github.com/StanAngeloff/vagrant-shell-
scripts](https://github.com/StanAngeloff/vagrant-shell-scripts)

~~~
801699
so it is the scripting language used by ansible, chef, puppet that imposes the
required "discipline"?

methinks each of these must in fact call the shell to get things done or least
system(3).

if they are calling execve then i would be more interested.

i would also be interested in ansible, chef, puppet if i knew the scripting
languages they are written in or wanted to learn them. but other languages
interest me more.

 _as a hobbyist_ , what i would like to see is a hosting provider that will
boot fs images (including bootloader) that user builds _on users own
computer_. i.e., provider gives exact specs of machine and network details.
user builds and sends fs image to provider and provider boots from it on some
bare metal in a datacenter.

no "host" or "guest". no virtual server. no provider software. if something
software-related does not work, it is user's responsibility because it is all
user's software. user can send an updated fs image.

anyone know if this exists? the service wanted is barebones: a computer in a
datacenter that has an internet connection and someone to boot it from user fs
image. cost not an issue.

~~~
lukaslalinsky
Ansible, Chef and Puppet are not languages. They are systems for managing the
state of your server, during the whole lifetime of the server. You typically
tell them "I want X to be in Y state" and they will take whatever actions to
make that happen. Writing all that logic in a shell script is not particularly
trivial. Of course, it's not all perfect and sometimes you have to resort to
scripting-like approach, but the idea is that for the common patterns, they
already have the logic implemented.

~~~
iso-8859-1
> You typically tell them "I want X to be in Y state" and they will take
> whatever actions to make that happen.

I had so many problems with this, because not all possibilities are tested on
all distributions on all architectures with all libc's and compiler
combinations. Much is implicit compared to Nix where only the kernel is
implicit.

~~~
CaveTech
Why would you be running on dozens of distros on different architectures?

With some sane level of standardization (lets say less than 6 combinations)
it's easy enough to test all and fix up the discrepencies.

The more declaritive you are, and the less you assume about the environment,
the better this will work.

Ie. if you need a package, always install it. On some distros it may be there
by default, but you'll run into issues if it's not. The good thing about being
declaritive is that there's no harm in checking.

------
dugmartin
Here is a declarative bash DSL I ran across a few months ago:

[https://github.com/mattly/bork](https://github.com/mattly/bork)

and here is an example from their README:

    
    
        ok brew                                       # presence and updatedness of Homebrew
        ok brew git                                   # presence and updatedness of Homebrew git package
        ok directory $HOME/code                       # presence of the ~/code directory
        ok github $HOME/code/dotfiles mattly/dotfiles # presence, drift of git repository in ~/code/dotfiles
        cd $HOME
        for file in $HOME/code/dotfiles/configs/*
        do                                            # for each file in ~/code/dotfiles/configs,
          ok symlink ".$(basename $file)" $file       # presense of a symlink to file in ~ with a leading dot
        done

------
edgan
This is just plain a bad idea. Bash is the worst possible language. Yes, it
can do it, but the syntax is awful. The problem of single quotes in double
quotes in escaped double quotes.

I replaced a home grown, written by someone else, bash configuration
management tool with Salt. The difference was huge.

Configuration management tools give you built-in logging, state checking,
return codes, template libraries, and access to very mature models to access
things like the AWS api.

I realize people don't like the steep learning curve, but there is a reason
for it.

------
atmosx
Honestly, I find ansible quite simple and straight forward. You don't have to
use third party playbooks and using 'autoenv' I achieved a very straight
forward, simple, dynamic setup.

~~~
mbubb
Agree - not sure what the advantage is. I understand bootstrapping can be a
minor irritation (and I dont have a great way to do it 100% auto)

Dynamic inventory scripts \+ ansible-playbook -k -u root ....

can do the same thing for bootstrapping.

How is this an advantage over that?

~~~
fredsted
With Ansible I use the 'raw' command to run a command to check whether Python
(dependency of Ansible) is installed, and install it if not.

    
    
        - name: Bootstrap Ansible
          hosts: all
          gather_facts: False
          tasks:
            - name: Install Python 2
              raw: test -e /usr/bin/python || (sudo apt -y update && sudo apt install -y python-minimal)

------
bjpbakker
Author claims all 90% of the time all they need is a single shell script. Then
shows an example directory structure that mostly reminds me of Ansible.

How is Welder different from Ansible, except bash vs python?

~~~
pavanky
It also depends on Ruby and liquid templates. I am not sure how he can still
claim it is "plain shell scripts"

~~~
tmzt
Bash has it's own substitution system built in, though I would usually use sed
for this task.

[http://tldp.org/LDP/abs/html/string-
manipulation.html](http://tldp.org/LDP/abs/html/string-manipulation.html)

See the section under Substring Replacement.

~~~
kasabali
Even M4 macros may work. It would be still more in the shell spirit than ruby
gems.

------
mikepurvis
I came from setup shell scripts (especially running in the postinstall step of
debian-installer) to Ansible, and I think the big win with Ansible is at least
the ideal of idempotency.

That's a lot harder to achieve with plain shell scripts, unless you're pretty
disciplined about only performing mutations to your system that are themselves
idempotent (like a package install).

~~~
blablabla123
Stupid question, but why are Ansible playbooks supposed to be "more
idempotent" than plain shell scripts?

Due to taking over a project I had to deep dive into it pretty quickly and to
me it feels like shell in yaml syntax.

~~~
GauntletWizard
The answer is, they're simply not. Ansible is just as easy if not moreso to
shoot yourself in the foot, only there's some smoke and mirrors (like their
'lineinfile' module) that make you think you're idempotent but actually just
create race-conditions and reduce version control.

There are a few tricks that Ansible uses that are non-obvious uses of shell -
For example, files should be replaced by copying the new version to the target
directory, then unlinking the original file and linking in the new version in
it's place, finally unlinking the temporary file. This functionality has been
implemented in the 'install' utility in gnu and bsd coreutils since the 80s
(though it's not in POSIX, so it's not exactly guaranteed; You'd be hard-
pressed to find a machine outside of busybox docker images that does not
support it)

~~~
icebraining
Race-conditions? As in, if you're running the same Ansible script concurrently
on the same machine? Yeah, definitively don't do that. But why would you?

I agree that modules like lineinfile are tricky and should be avoided (better
to copy a full file), but it's hardly just as easy to shoot yourself than the
equivalent sed command, in my opinion, if nothing else because they're more
restricted.

------
CyberShadow
I created something similar for Arch Linux:

[https://github.com/CyberShadow/aconfmgr](https://github.com/CyberShadow/aconfmgr)

It has a few crucial differences from typical configuration managers, though,
as it allows transcribing the system state back into your configuration.

------
oneplane
Basically, this is SaltStack's salt-ssh re-implemented using bash instead of
python... That too is just YML + Templates + Bash + SSH, with the exception
that you can use Python too and the templates are JINJA.

------
callumjones
If you don't like all the directories that Ansible playbooks require you can
use the simple style of a single YAML file that describes the actions to take.
Check out [https://serversforhackers.com/an-ansible-
tutorial](https://serversforhackers.com/an-ansible-tutorial), the first
introduction walks you through setting up just 2 files.

------
mariocesar
I did something Similar, but fully based in shell scripts with no dependencies
->
[https://gist.github.com/mariocesar/8e674ec40dad6b94114d2a44d...](https://gist.github.com/mariocesar/8e674ec40dad6b94114d2a44d9151e23)

I completely agree with OP when Ansible is overkill for must cases, and even
slower than a plain smart shell script.

For my solution, simple create a bash function:

    
    
      function play ()
        local remote=${2}
        local script=${1}
        local directory=$(dirname ${script})
        tar cpf - ${directory}/ | ssh -t ${remote} "
          tar xpf - -C /tmp &&
          cd /tmp/${dirname} &&
          bash /tmp/${script} &&
          rm -rf /tmp/${dirname} "
      }
    

and later just call it like.

    
    
      play provision/bootstrap.sh ubuntu@10.0.0.1
    

Where provision is a directory, and bootstrap.sh is the main shell script.

------
joelcollinsdc
Reminds me of
[https://github.com/brandonhilkert/fucking_shell_scripts/blob...](https://github.com/brandonhilkert/fucking_shell_scripts/blob/master/README.md)

------
ausjke
I was really excited until I saw "Welder requires rsync, ruby and liquid" \---
why do I need Ruby and all its dependencies here?

I tried
[https://github.com/myplaceonline/posixcube](https://github.com/myplaceonline/posixcube)
and it has zero dependencies , just bash itself, and it is nice.

------
cat199
On the subject of non-approved sysadmin/devops tools,

I wrote a little 'tarball compiler' framework in mostly PMake some time ago to
"DevOps" like it's 19[89]9 ... basically it will build per-host file overlay
trees, rdist them out to target hosts, and then optionally run user defined
start/stop/restart hook scripts when needed.

[https://github.com/ixcat/admmk/blob/master/doc/admmk.rst](https://github.com/ixcat/admmk/blob/master/doc/admmk.rst)

unfortunately, though it will do per-host targets, I didn't get around to per-
service targets.. should be feasable however.

Definitely a bit hairy, but hey, thats what quasi-functional symbolic makefile
programming is all about my friends..

ps: if you think this is bad, try rewriting it in gnumake if that's even
possible.. PMake! Woo!

------
nwmcsween
cdist was pretty much the same, you should see why they made the decisions
they made

------
nodesocket
There is really not much you can't do using HashiCorp's Packer[1] and
Terraform[2].

Packer can generate base images and roles based on shell scripts and
transferring files. Terraform manages the creation of infrastructure on clouds
and can bootstrap instances with shell scripts as well.

[1] - [https://terraform.io](https://terraform.io) [2] -
[https://packer.io](https://packer.io)

------
jazoom
Sup also works well

[https://github.com/pressly/sup](https://github.com/pressly/sup)

------
snissn
Could you explain what this line of code does?

    
    
        ssh -t user@example.com "$(< ./my-setup-script.sh)"

~~~
falsedan
It's a weird way of writing

    
    
      ssh -t user@example.com /bin/bash < my-setup-script.sh
    

That is, take the contents of the local file 'my-setup-script.sh' and run them
as commands on example.com as the 'user' user (using SSH to authenticate the
user & transport the commands).

~~~
iso-8859-1
But that might fail half-way through the file, and the other one is all or
nothing, might just report "argument list too long".

~~~
falsedan
> _might fail half-way through the file_

Oh, while transferring!

    
    
      ssh -t user@example.org bash <( echo "function do_it() {"; cat my-setup-script.sh; echo -e "}\ndo_it" )
    

No argument limits to worry about, no quoting hassles.

------
benbristow
I was about to make the point about Ansible but I guess you've already made it
in the README.

~~~
giobox
Is Ansible really too much though, as the author suggests in README for simple
tasks? If all you want is basic "SSH to a server, install some packages" type
behavior, I'd still make the argument Ansible is simpler than this. Ansible
isn't like Chef/other competitors where you have to bootstrap or install a
client on the target machine. IIRC it only really needs SSH access and a
Python installation present on the target machine, which is basically most
Linux distros.

Looking at how you use this thing, I'm really not sure it would save me any
time versus using Ansible.

~~~
marmaduke
There's also a learning curve case to be made against Ansible: if you've spent
years polishing your Bash and coreutils one liners, you start over at zero
when switching to Ansible.

Ansible is also declarative (in principle) while shell scripts are imperative,
which is a bit of mind bender. Then, when you debug, you have to understand
that Ansible is implemented by imperative code.

~~~
zandor
The same could be said for some NIH tool.

Bringing people up to speed on in-house tools can be a major hassle. Never
mind the fact that popular tools like Ansible are battle-tested and most
likely has more documentations and tutorials.

~~~
marmaduke
the link here was for a tool using just SSH and Bash..?

------
schizoidboy
Similar tool:
[https://github.com/myplaceonline/posixcube](https://github.com/myplaceonline/posixcube)

------
yakshaving_jgt
Similar: [https://github.com/kenn/sunzi](https://github.com/kenn/sunzi)

------
ajmarsh
Hm I've always just used pssh for this type of administration. I guess this
saves a step not having to pscp.pssh the script onto the nodes.

------
nikolay
Yet another Ruby-based solution?!

~~~
dragonshed
Yeah, I also was a little disappointed with the dependencies, but the general
approach has merit.

