
Show HN: Ultimate Plumber – a tool for writing Linux pipes with live preview - akavel
https://github.com/akavel/up
======
ptman
Careful with rm:

    
    
        r
        rm
        rm -
        rm -rf
        rm -rf ~
        rm -rf ~/
        rm -rf ~/tmp
        # where did my files go?

~~~
somanyquestions
yikes!

It's always been surprising to me that their isn't a built in "undo" to `rm`

e.g. why doesn't `rm` just move files to /tmp?

~~~
clarry
> e.g. why doesn't `rm` just move files to /tmp?

And do what when tmp gets full and services & applications start crashing?

When someone implements that undo, someone's going to need to write a tool
that really does remove files, really. Will the next guy wonder why really-rm
doesn't have undo built in?

~~~
diegoperini
What about undo that expires in 1 minutes? Gmail website does that for sent
emails (for 5 seconds I guess).

~~~
hk__2
> What about undo that expires in 1 minutes? Gmail website does that for sent
> emails (for 5 seconds I guess).

Same question as parent: "And do what when tmp gets full and services &
applications start crashing?"

~~~
kungtotte
You could do what is more or less an industry standard when it comes to this
and warn the user that there's not enough room to allow an undo when deleting
said file(s) and prompt if they want to delete it permanently instead.

It's not rocket surgery.

------
jlv2
One of my favorite quotes:

``Remember, a temp file is just a pipe with an attitude and a strong will to
live'' \- Randal Schwartz [comp.unix.questions]

~~~
joncrane
I never thought of it that way. Framing it that way is ingenious.

~~~
JdeBP
pipes _used to be_ temporary files on Unix.

* [https://unix.stackexchange.com/a/450900/5132](https://unix.stackexchange.com/a/450900/5132)

------
pjungwir
This looks like a fun and helpful tool! (Perhaps best to ignore the notice
from lshw and _not_ run it as root, ha ha. Maybe even only in some kind of
sandbox. I bet they could add a --safe flag so it would make an on-the-fly
Docker image with a copy of your cwd, and run the commands in there. Then you
have different problems I suppose, but still....)

I've never seen `paste - -` before, but that is amazing too! I can't count how
many times I've wanted something like that. Surely that will help my own shell
pipelines in the future.

Also I've never seen `foo |& bar` before. What a neat idea! The surprisingly-
valid creativity of it reminds me of Duff's Device. Has anyone else ever found
uses for that idiom?

After so many years working in the shell, it's such a rare and delightful
treat to learn new tricks. Thank you for sharing this with us!

~~~
DanielRibeiro
It is a pretty neat trick. Kinda hard to Google for though. Here is the bash
manual page on it:
[https://www.gnu.org/software/bash/manual/html_node/Pipelines...](https://www.gnu.org/software/bash/manual/html_node/Pipelines.html#Pipelines)

------
pareshverma91
Looks great. Personally, I have been using a "cache" function to cache output
of expensive commands. This has helped me iterate faster on pipelines that
call into apis.

    
    
      function cache {
          cmd="$*"
          name=$(echo "$cmd" | base64)
          if [[ -f ./cache/$name.exit ]] && [[ $(cat ./cache/$name.exit) -eq 0 ]]; then
              echo "cached"
          else
              mkdir -p ./cache
              eval "$cmd" > ./cache/$name.out
              echo $? > ./cache/$name.exit
          fi
          cat ./cache/$name.out
      }
    
      cache ls /

~~~
zbentley
I use something similar, created by a friend of mine, called 'bash-cache':
[https://bitbucket.org/dimo414/bash-
cache/src/default/](https://bitbucket.org/dimo414/bash-cache/src/default/)

It hooks specific functions in Bash and keys off of their arguments.

------
jxy
I do progressive filtering a lot. It's convenient to have all the intermediate
results somewhere temporarily.

What I do is typing the command in one acme window with the working directory,
and middle mouse swipe it. Acme shows the result of that command in a +Errors
window with the same working directory. I change the name of this new popped
up window (to something like `+Errors-cmd`, type a command that operates on
the file name `/mnt/acme/$winid/body`, and middle mouse swipe it again. Acme
shows the result of the new command under `+Errors` again. Rinse and repeat
until I'm satisfied.

I guess this way it gives a convenient temporary place for all the
intermediate texts. I can of course edit them in place if I want.

~~~
duck2
A rare sight, another user of Acme on HN!

You know it probably, but there is also a mouse chord for piping selected text
into a command. (2-1)

~~~
jxy
I was curious what this plumber could be.

2-1 is just a concatenation of two strings and run the whole string. I
wouldn't call it pipe.

------
esotericn
This is awesome, but I am absolutely bloody terrified of using it on my
systems.

As is stated in the README, if you write 'rm' or anything like it... oopsie
oops.

~~~
jolmg
Right, that's one problem. The author says:

> But you'd be careful writing "rm" anywhere in Linux anyway, no? Also, why
> would you want to pipe something into "rm"?

But the thing is that you don't need to intend to pipe to "rm". Maybe you were
typing something else, like having a command "rmore" or something. This danger
is also not strictly limited to `rm` and `dd`, and you have to be careful that
no substrings from the start of the command you intend to write cannot be
interpreted as something dangerous.

~~~
wpietri
The author's theory there about user behavior seems dangerously wrong.

I'm quite careful writing command lines because the whole experience of the
shell is that of working with sharp knives: you get a lot of power but if you
screw up, you'll feel the pain. The point of this tool is to take away a lot
of the pain that teaches people caution. Its whole theory is "just try a lot
stuff as you explore what you want".

In their shoes I'd look at using some of the container/security magic as a way
of nerfing commands. If the on-a-keypress runs work in a way where they can't
make changes to a filesystem, that seems way better to me. Even better if the
tool then reports, "Would have deleted 532 files in a warning color at the top
of the output."

~~~
groestl
> I'm quite careful writing command lines

A safety measure I picked up from a sysadmin while watching over their
shoulder: start writing nifty command lines by prefixing them with # first, to
prevent havoc when fat fingering the [enter].

~~~
yread
It's a pity you can't do something similar to SQL: when writing UPDATE/DELETE
always write the WHERE clause first

~~~
rmetzler
Something I learned early on when writing possible dangerous SQL selects:

    
    
      CREATE TABLE mytable_backup AS SELECT * FROM mytable;
      SELECT * FROM mytable WHERE condition;
      DELETE FROM mytable WHERE condition;

~~~
traviscj
Or just let the DB work for you with BEGIN; + (select/update/delete) + COMMIT;
or ROLLBACK;

------
ElijahLynn
After watching the GIF on the README.md I was like "WHERE HAVE YOU BEEN ALL MY
LIFE" (actually just the past 10'ish years)!!!

Then I was like, I bet quite a few ancient wizards out there have some pretty
amazing techniques they have picked up over the year. And that it would be
great to be a "fly on the wall" watching some of these wizards do simple
tasks. And that a YouTube channel watching them do simple tasks could be quite
revealing and therefore very educational.

Paul Irish did something like this a few years ago and I still use a bunch of
the tools he introduced into my purview. Really, really improved productivity!

~~~
catacombs
> And that a YouTube channel watching them do simple tasks could be quite
> revealing and therefore very educational.

Does such a channel exist? All I've ever found has been tutorials. I want to
watch unix veterans use the tools. I've found that to be the best way to
learn. Hell, that's how I learned vim.

~~~
kapad
which channel did you use to learn vim?

~~~
catacombs
No channels specifically. I've watched these videos as a good primer:

[https://www.youtube.com/watch?v=qsWY-8n9igM](https://www.youtube.com/watch?v=qsWY-8n9igM)

[https://www.youtube.com/watch?v=_NUO4JEtkDw](https://www.youtube.com/watch?v=_NUO4JEtkDw)

[https://www.youtube.com/watch?v=blpCz9HsUaA](https://www.youtube.com/watch?v=blpCz9HsUaA)

------
traviscj
Nicely done!

I do a pretty similar version of this with command-| in textmate, but I've
wished for/thought about building a command like this to speed up the process!

The textmate approach does have the advantage (or perhaps _dis_ advantage, for
large enough inputs!) of emitting immutable "views" of the data at each step,
rather than (presumably?) re-evaluating the pipeline each time, which is nice
if the steps of the pipeline are the slow part, or to go back 2 steps and
start a new pipeline. Maybe a potential future flag? :-)

Automatically emitting an `up<N>.sh` script is super clever too!

~~~
akavel
Can you please elaborate a bit on the textmate approach? I don't fully grasp
it from this description yet... while it totally sounds interesting...

~~~
traviscj
I (partially) recreated your `seq 1 15` example here [1] -- no sleep, but my
lshw wasn't as interesting as yours :-) I'm hitting command-shift-| to bring
up the "filter through command" dialog.

[1]:
[https://tcj.io/i/textmate_piping.gif](https://tcj.io/i/textmate_piping.gif)

~~~
akavel
Ok, I get it now. Thanks! :)

------
_kst_
I've submitted an issue on GitHub, title "Executing incomplete commands is too
dangerous".

[https://github.com/akavel/up/issues/8](https://github.com/akavel/up/issues/8)

------
daenz
Really cool. The key principle that this tool highlights is the advantage of
rapid iteration. Whenever you have a problem, spending the time up front to
shorten your test iterations will pay off massively in the long run of the
problem's life.

------
the8472
This is nice for linear pipes. But occasionally I find myself building more of
a graph.

Bash lets you provide file arguments from pipes via the <() syntax (it's
turned into a fd reference via proc behind the scenes). And you can also wire
up additional pipes with fd renumbering, though this gets ugly in bash, some
other shells are more flexible there, e.g dgsh.

A tool to iteratively build a graph of commands and pipes would be the next
step.

~~~
mohammedbin
This sounds interesting. Can you pull up some history and make a gist/pastie?

------
undershirt
I checked out the original tool Pipecut—it allows you to look at the output
after each pipe:

[https://youtu.be/CZHEZHK4jRc?t=1273](https://youtu.be/CZHEZHK4jRc?t=1273)

When I see stuff like this, I check the date to see if it happened before or
after "Inventing on Principle" was presented in 2012. (It's always after)

~~~
akavel
Hmmm, you made me think and try to recollect. And to my surprise, I can't
really _remember_ the fact of The Presentation being an influence. But I am
quite sure _it must have been_ , I can see no other option. Hm, memory is a
tricky thing. And Bret Victor left his imprint on our whole industry.

------
d0mine
Related: percol utility allows to narrow a process output interactively
[https://github.com/mooz/percol](https://github.com/mooz/percol)

I use it all the time to select a command from my shell history. It is a very
convenient addition to zsh-autosuggestions (one of those things that you use
hundreds times per day without even thinking about it).

------
ddalex
Late to the party. This screams like it needs to be part of the shell instead
of an addition.

E. G. I type a command in the shell, end it with pipe, and press enter, it
doesn't delete the command but runs a preview of the output top 10 lines? This
way I can continue to the edit the command without losing the context, the
pipe, or I can decide to redirect it to a file!

~~~
akavel
Hmm, interesting thought! The tool needs to do some ugly buffering however;
would there be place for it in a shell? Also, I suppose some kind of "Ctrl-
Enter" or something would be a better idea instead? So that it could also be
hit at any point in the line... But how about scrolling through the preview?

------
sisk
This is great. Thanks for sharing.

For something a lot dumber: here's a tiny thing to print out pipe contents to
allow for cancelation before continuing the pipe.

[https://gist.github.com/jasisk/be34ae93b74a3d3e8f0a4daac1237...](https://gist.github.com/jasisk/be34ae93b74a3d3e8f0a4daac1237966)

------
gregwebs
The most frequent use case for this for me would be interactive grep. The
"fzf" tools has this along with fuzzy search. And the text doesn't disappear
while you type "grep" like it does with "up".

I will definitely try this out too though when I make a pipeline with sed,
cut, etc.

~~~
isodude
You know that less has interactive grep right? Type & phrase <enter> To go
back: & <enter>

------
miduil
Just an awesome tool, I love those moments of "ah that's trivial, why didn't I
think of this already" and it definitively applies for this tool. So special
kudos for the idea, but also how little go code the implementation took
(ignoring dependencies).

There is already some "pre-release" discussion going on over at lobsters btw.

[https://lobste.rs/s/acpz00/up_tool_for_writing_linux_pipes_w...](https://lobste.rs/s/acpz00/up_tool_for_writing_linux_pipes_with)

~~~
akavel
Thanks! :) See also my "prior art" note in the readme ;) I also had the "why
didn't _somebody_ think of this already" moment when _I_ thought of it ;) And
somebody in fact more or less did, just I wasn't able find it then :)

------
amarshall
Reminds me of fzz [0], which is a similar tool but focused more on interactive
modification of prescribed components of a known command/pipeline than
appending a pipeline.

[0] [https://github.com/mrnugget/fzz](https://github.com/mrnugget/fzz)

------
Too
Novel and interesting idea but honestly i can't see any other use for this
than grep and that point you might as well just pipe the output straight into
vi, less or your other favourite editor where incremental search is one
keystroke away.

Sure, the shell has some powerful tools like jq that editors usually lack but
otherwise several levels combo piping through cut, wc, awk, etc are quite rare
and are usually only done in crazy shell scripts that actually should be
rewritten in more appropriate language.

And the dangers of automatically running any commands shouldn't be downplayed.
If you only whitelist a few programs this would be much safer, most users
would only use this for grep. Or run the whole thing through a docker
container.

------
dustfinger
rm is not the only issue. Another problem may arise if your command is not
idempotent with respect to the state of the system. Especially if it is not
idempotent with respect to system state and requires multiple parameters,
since it will execute many times as the parameters are typed. Maybe having a
configurable debounce on key press with a reasonable default would be
sufficient to get around this issue.

What would be really cool is if this tool ran in a pretend mode and showed you
what the results would be without actually mutating the system. Then, once the
user is happy with the results, the command can be executed for real.

This whole concept reminds me of helm for emacs :-). Thanks for sharing!

~~~
akavel
Thanks for the ideas, and for the good words! :) Do you know of any nice
screencast/video where some similar functionality of helm would be shown? Does
it also work for shell commands in some way? I'm not an emacs user...

------
nailer
> This is achieved by boosting any typical Linux text-processing utils such as
> grep, sort, cut, paste, awk, wc, perl, etc., etc., by providing a quick,
> interactive, scrollable preview of their results.
    
    
        lshw | grep network -A2 | grep : | cut -d: -f2-
    

A better idea would be to work on tools that have structured output, so people
can select the keys they want and don't have to scrape.

    
    
        $ Get-NetAdapter | where status -eq 'up' | select InterfaceDescription
    
        InterfaceDescription
        --------------------
        Qualcomm Atheros QCA61x4A Wireless Network Adapter

------
arendtio
Sounds like fun, but I think there are other ways to achieve a similar
workflow without an extra tool. For example, I like to use a tmux session with
two panes. On the one side there is a `vim myscript.sh` and on the other side
there is a `watch myscript.sh`.

Very similar result and using rm is not that dangerous as you can finish
writing your code and it gets executed as soon as you save the file (if you
don't like the watch delay you can use one of the many inotify based watch
tools or set -n 0.1 or something that suits your needs).

~~~
theossuary
I use `entr` for a similar effect, little nicer than watch, though I don't
think it's available on all OSes.

------
codetrotter
Well damn! This is a really elegant solution for a problem that has been
bothering me for quite a while, and for which my own planned solution was much
more complex :O

~~~
akavel
Thanks a lot for the good words! :D <3

------
mohammedbin
Why wasn't there a tool for this earlier? I am a very aggressive piper but I
used to hit up arrow , a series of ctrl-w and enter and wait everytime.

Thank you

------
jlv2
One of my favorite quotes:

``Remember, a temp file is just a pipe with an attitude and a strong will to
live'' \- Randal Schwartz [comp.unix.questions]

------
ygra
One thing that probably would be useful is when commands are not completely
typed or parametrized at the end of the pipeline to still show the last result
along with the error message instead of replacing everything and thus removing
any context the user had.

------
JdeBP
* [https://github.com/akavel/up/blob/master/up.go#L574](https://github.com/akavel/up/blob/master/up.go#L574)

Please at least respect the SHELL environment variable. (-:

~~~
akavel
Eheh; sorry, MVP! ;) and thanks a lot for taking it lightly and commenting in
such a kind and playful way :D It's totally on the long list in some way...
please see
[https://github.com/akavel/up/issues/2](https://github.com/akavel/up/issues/2)
! :)

------
spark888
There is a vim plugin that enables "live preview" for python :
[https://github.com/metakirby5/codi.vim](https://github.com/metakirby5/codi.vim)

------
jspears
What about using LD_PRELOAD to load a FS shim that prevents writing to the FS?

~~~
akavel
I don't think LD_PRELOAD would be enough; there are programs which don't use
libc and reach straight for syscalls (e.g. the whole Go ecosystem/runtime)

~~~
jspears
Sure; it wouldn’t be perfect safety but would work for a large swath of Unix
tools.

------
swiley
Maybe if it automatically chrooted everything to prevent side affects (or
better yet: isolate them to a ramdisk.) That would be awesome.

------
agumonkey
Massive bravo. Thought about the need for this for ages.

------
msravi
For mac: go get -u github.com/akavel/up

------
newnewpdro
Wow, how reckless! running partial commandline constructions on every
keystroke..

The obvious thing to do is only run what's been typed so far on a trigger key
like tab or something.

------
amarant
this looks great! installing! my one gripe is that the name really should have
been marIO ;)

