
Show HN: Using Rust to write shell-script like tasks - rustshellscript
https://github.com/rust-shell-script/rust_cmd_lib
======
jitl
I love this. I tried to do something similar in Go, because it was in use on
my team at Airbnb, and we were looking to port a 2000 line make-and-bash tool
to... something not make-and-bash. But as you know, Go doesn’t have macros -
so I spent all this effort trying to build a @decorator comment macro system
in my personal time (abandoned). Rust seems like a perfect fit for this! We
did have a teammate pitching rust, but no one wanted to learn it.

Anyways, congrats on the release. It looks fabulous. Safely splicing shell
command snippets together is surprisingly annoying, so it’s really cool to see
a hygienic yet user-friendly approach.

------
gitgud
Interesting, it seems that it allows the exact syntax of shell commands,
without using strings.

    
    
        // valid rust code and shell code, no strings
        run_cmd!(du -ah . | sort -hr | head -n 10)?;
    

How does rust parse the statement within _run_cmd()_? Can rust parse other
languages like this?

    
    
        run_html!(<div>COOL</div>)

~~~
steveklabnik
[https://crates.io/crates/typed-html-macros](https://crates.io/crates/typed-
html-macros)

------
rhn_mk1
I am not a huge fan of copying the shell language wholesale and wrapping
inside a macro. Since macros can execute arbitrary code, this makes me feel
uneasy that the strings are just executed within a shell context, with all the
appropriate, bug-prone, expansion done by the shell.

Seeing "ls /nofile || true;" makes me worry that "||" is actually passed to
the shell wholesale. There's also no transparency about how the binary names
are resolved.

I much prefer an approach more integrated with the language, like Plumbum:
[https://plumbum.readthedocs.io/en/latest/local_commands.html...](https://plumbum.readthedocs.io/en/latest/local_commands.html#guide-
local-commands)

This no longer looks like the POSIX shell, but instead clearly integrates the
good parts directly into the language, even if some complexity bubbles
through. I don't have to worry that "grep["world"] < sys.stdin" is piped into
an actual shell, because it gets converted into an AST on the way to
execution.

~~~
papaf
_Since macros can execute arbitrary code, this makes me feel uneasy that the
strings are just executed within a shell context, with all the appropriate,
bug-prone, expansion done by the shell._

Its a shame you didn't bother to look at the source code before criticizing.
Someone put a lot of work into this library and its actually pretty cool.

The package parses the code in the macros [1] and then calls
'std:Process::Command' [2] which, I believe, does not execute a subshell by
default.

[1] [https://github.com/rust-shell-
script/rust_cmd_lib/blob/maste...](https://github.com/rust-shell-
script/rust_cmd_lib/blob/master/crates/cmd_lib_core/src/parser.rs#L149)

[2] [https://github.com/rust-shell-
script/rust_cmd_lib/blob/maste...](https://github.com/rust-shell-
script/rust_cmd_lib/blob/master/crates/cmd_lib_core/src/process.rs#L313)

~~~
viraptor
I don't think the parent said this is not parsed well, or at least I didn't
read it that way. I share the feeling that you see that code and unless you
know the implementation it's not clear what shell brokenness is carried over
and what isn't. And which shell and version is being emulated. It's much
easier to set expectations with a new syntax that's also easier to document
than "what to expect of this macro".

~~~
rhn_mk1
Thank you, that's indeed what I meant.

The other, related issue is that reimplementing pieces of the shell DSL
duplicates what can be done in the parent language.

Taking conditional return value as an example: "ls /nofile || true;"

In this case I don't really want to be given the option to use bash syntax for
this. That would encourage the usage of shell idioms for "tricks" like control
flow, which are another annoying part of the shell (I can never remember
them). I would much prefer if there was a nice way to do that kind of things
idiomatically in the parent language, and no other choice. E.g. to ignore the
return value I would find it much nicer to be forced to do sth like this
instead:

let _ = ls('nofile');

~~~
rustshellscript
since run_cmd! and run_fun! are returning result type, you can always do let _
= run_cmd!(ls nofile); to ignore single command error.

The “xxx || true” is for ignoring error within a group of commands, which is
also very common in sh “set -e” mode. Without it, the group of commands need
to be divided into at least 3 parts to still capture all possible command
errors. I probably need to document this part with more details.

~~~
rhn_mk1
This ties in to what I wrote before: I prefer a philosophy where groups of
commands are not written in the shell DSL, but are instead native statements
(as much as possible), and the user is forced to use native control flow.

Documentation is not going to make me warm up to the idea, because I don't
like having the choice to use the DSL so much.

With that in mind, perhaps I'm not the most valid person to provide criticism
of this project ;)

------
yisonPylkita
My god, this super useful when you have a mix o shell commands and processing
text output from them. Bash isn’t particularly easy to work with parsing non-
trivial strings in a readable way (I’m looking at you awk)

~~~
ilovetux
I apologize, but I must disagree. Awk is literally amazing once you get used
to writing actual scripts instead of trying for the ever-elusive and often-
untenable one-liners.

Noone is using `python -c` syntax for constructing one-liners and i think
thats helping adoption of python. I have no idea why one-liners are seen as
desirable when they're often hard to read and debug.

~~~
qchris
When I first was getting into coding while doing test engineering, this was
one of my greatest complaints. The software engineers would hand me bash
scripts filled with very clever, but unexplained and esoteric one-liners.

Whenever something didn't work (and it didn't, because perfectly interfacing
with embedded hardware is tough), I had two options: spend literal hours on
Google and Stack Overflow, or go stand outside their cubicle and hope they had
time for me.

I'll take a verbose function with clearly-followable logic over an amazing
one-liner with a maze of options and hacks any day.

------
oconnor663
I'm going to shamelessly plug my own library here:

[https://github.com/oconnor663/duct.rs](https://github.com/oconnor663/duct.rs)

I wanted to solve the same problem, originally in Python
([https://github.com/oconnor663/duct.py](https://github.com/oconnor663/duct.py)).
It's surprisingly annoying to do pipelines and redirections, compared to how
easy they are to do in the shell. Lots of libraries try to address this, but
most of them seem do it by emulating shell syntax within the host language,
using operator overloading or other magic like that. I think that's a limiting
choice. (For example, can you use `cd` to change the working dir for the left
half of a pipeline but not the right half? In Bash you would use a "subshell"
for this.) Instead, I think it's sufficient to build an API out of regular
objects with regular methods. The result doesn't look like shell code, but
it's easier to reason about, and more consistent across different languages.

~~~
rustshellscript
It can be supported with internal APIs, even without macros:
Cmds::from_cmd(Cmd(...).current_dir(..)) .pipe(Cmd(...).current_dir(...))
.run_cmd(...)

As you can see, it is very verbose and that's why I choose to hide the lower
APIs at this moment.

~~~
joobus
I like your approach more than duct.rs :)

------
geowwy

      > A lot developers just choose shell(sh, bash, ...) scripts for such tasks,
      > by using < to redirect input, > to redirect output and '|' to pipe outputs.
      > In my experience, this is the only good parts of shell script.
    

If you try to use shell as a general purpose programming language, of course
it sucks.

If you treat shell as a DSL for files and streams, nothing can beat it. Shell
is amazing.

I'm sceptical a bunch of Rust macros can beat shell. I think you'd better off
writing a few smaller programs that use STDIO and stringing them together with
shell.

~~~
masklinn
> If you try to use shell as a general purpose programming language, of course
> it sucks.

> If you treat shell as a DSL for files and streams, nothing can beat it.
> Shell is amazing.

The problem is that any non-trivial shell script is a mix of the two, so you
find yourself torn apart by the inconvenience of "file and streams" in most
languages (though really it's mostly subprocesses), and the inconvenience of
_literally everything else_ in shells.

~~~
amalcon
This is the one space where I actually like Perl: for shell scripts that grew
up a bit, but still amount to mostly manipulating text files and streams.

------
LockAndLol
This reminds me of the python version of this called xonsh
[https://xon.sh/](https://xon.sh/)

I really like the idea, but it was missing some simple features that bash had.
I can't recall them right now but after an hour of trying to convert a simple
bash script, I gave up. That was a year ago. Maybe things changed.

I'll give this and xonsh a go again because I just really dislike bash. Thanks
for the project!

~~~
oldsj
I’ve been playing with xonsh lately and really liking it so far! From what I
can tell it’s pretty close to feature parity with fish shell which has a lot
of nice things like command auto completion but you don’t have to learn yet
another shell syntax it’s just python. Wrote up a quick trip report at
[https://blog.jamesolds.me/post/xonsh-aws-
example/](https://blog.jamesolds.me/post/xonsh-aws-example/)

------
rossmohax
Nothing can fix the fact, that pipes carry dumb byte streams. Powershell
addressed this, but sadly remains unpopular in Unix crowd

~~~
hnlmorg
Actually there are several shells out there that fix that problem _and_ still
support existing UNIX tools too (which Powershell doesn't play nice with).

My own shell, [https://github.com/lmorg/murex](https://github.com/lmorg/murex)
does this by passing type information along with the byte stream. So _murex_
aware tools can have structured data passed and POSIX tools can fall back to
byte streams. Best of both worlds.

The problem, however, is that as long as Bourne Shell and Bash are installed
everywhere, people will write scripts for it. This is less about the
popularity of UNIX tools and more about the ubiquity of them (though the two
points aren't mutually exclusive).

~~~
carlmr
>The problem, however, is that as long as Bourne Shell and Bash are installed
everywhere, people will write scripts for it.

This is also an issue of interpreted languages. Often I write bash and very
constricted python2/3 compatible code, because I can be fairly sure the target
audience has both of these.

You need to have everyone install (and maybe even use) your shell/language for
them to be able to use it. Or have them recreate your environment (docker or
cxfreeze). With Rust it's easy to distribute a small self contained binary.

------
staktrace
I wrote a tool to do the opposite thing: allow writing "shell scripts" using
Rust. Still early days but
[https://github.com/staktrace/khaki](https://github.com/staktrace/khaki) is
where it lives.

------
Fiahil
This is great!

What about "set -euxo pipefail"? I see you were using eprintln before the
commands, could we have a "set" macro that would do that for us?

~~~
rustshellscript
You can consider it was enabled by default, and any failed command would
return error unless you mask it with “xx || true”.

------
lugoues
There is also [https://github.com/igor-
petruk/scriptisto](https://github.com/igor-petruk/scriptisto) which would
allow you to wrap any compiled language.

------
implfuture
This is so awesome! Any tips on parallelizing/is it async compatible? I
personally found granular error handling combined with parallelization to be
impossible to get just right in pure bash.

------
sneak
Does anyone know of something similar for Go?

------
ausjke
what's the difference between this approach and shell scripts? Thanks.

------
barrenko
I really don't see the point of these. Probably just need to explore available
tools a bit more.

------
darthrupert
If this is a fun proof of concept, it's nice.

If somebody uses this in an actual system, it's terrifying.

 _edit_ oh, Rust is now a thing where even the bad ideas need to be praised
without caveats. Gotcha

~~~
MetaDark
I don't think you deserve to be down voted for this. You bring up a valid
concern. Although, I don't agree that this project should be outright
dismissed either.

So many times, I've run into the issue where I've wanted to chain a set of
commands with a concise syntax (specifically in Python) without having to
shell out to bash.

What I really like about this library is that it gives you the concise
composability of bash, without having to deal with its pitfalls (eg. variable
escaping, lack of Windows support, clunky interface for anything that's not a
command invocation...).

Using a DSL will always come with certain tradeoffs, and it won't be the best
solution for every use case, but I think this library fills a certain need
very well.

------
laumars
Please people, don’t do stuff like this for anything other than personal
projects. You might think it’s safer than writing Bash but it isn’t.

It results in unsafe Rust code since you’re now forking external code that
might be missed by people who are strictly vetting for code inside “unsafe”
blocks. Ironically anyone who writes she’ll scripts will know that there are
problems with shell scripting but thankfully dot-sh files stand out and bring
attention to themselves as files that need to be audited. This wouldn’t. If
you need to embed other languages or even just the approximate concept of
then, then please at least keep those language files separate rather than
inlining them.

Then you have an issue that people who are already aware of the pitfalls of
shell scripts would know to read through any such scripts but this introduces
a newer and unfamiliar scripting language to audit (eg how do we knew that
what’s been declared is run but free?). At least Bash et al has had many years
of eyeballs on it.

~~~
kazinator
> _unsafe Rust ... since you’re now forking external code_

Are you saying that Rust becomes unsafe because it used a C program as a
subroutine? E.g. "tar xvf -" or whatever? What is the fix: rewrite tar, awk,
scp and whatever else as Rust functions? That's a lot of work.

I'm surprised that you're simultaneously overlooking what ought to be a more
gaping problem: that every system call made by a Rust program is a trip
through a kernel written in C.

~~~
intc
Could you please be more specific how it's a "gaping problem" that the
underlying kernel is written in C? I think even you'd write a pure Rust kernel
from scratch it would take a considerable time to achieve the same
quality/performance ratio as we are currently witnessing with C based kernels
(*BDS & Linux). It's so easy to throw these "radical claims". Yes? =)

~~~
feanaro
I think the idea is that if calling external C binaries is a problem, then a
kernel written in C is an even larger problem. It was meant as reductio ad
absurdum.

