
Show HN: Pyp – Easily, magically, run Python at the shell - hauntsaninja
https://github.com/hauntsaninja/pyp
======
j88439h84
Mario is a very featureful application of similar ideas, and includes multiple
built-in transformations, concurrency support, user-defined commands, and
extensive documentation.

[https://github.com/python-mario/mario](https://github.com/python-mario/mario)

~~~
hauntsaninja
Thanks for sharing! I hadn't seen Mario before. The async support is super
nifty! From a quick look, it looks like pyp might be a little quicker for some
use cases since it doesn't need subcommands, but I'll definitely be updating
[https://github.com/hauntsaninja/pyp#related-
projects](https://github.com/hauntsaninja/pyp#related-projects)

------
chrisweekly
My cousin helped create xonsh ([https://xon.sh](https://xon.sh)), a POSIX-
compliant "python as drop-in shell replacement", years ago.

~~~
hauntsaninja
xonsh is great! I'm a fan of your cousin's work :-) I mentioned it in
[https://github.com/hauntsaninja/pyp#related-
projects](https://github.com/hauntsaninja/pyp#related-projects) for if you
want even more Python in your shell.

------
jeswin
Nice. I made something similar, but for JavaScript
[https://github.com/jeswin/basho](https://github.com/jeswin/basho)

It's very useful to be able to automate with a language you already know. I
use it quite a bit now, for instance to figure out which of my git repos need
a push or a pull.

------
gmfawcett
This made me nostalgic for a utility I wrote back in 2005 called `pyline`:

[http://code.activestate.com/recipes/437932-pyline-a-grep-
lik...](http://code.activestate.com/recipes/437932-pyline-a-grep-like-sed-
like-command-line-tool/)

I'm sure I wasn't the first to scratch that particular itch. It always seemed
odd, back then, that Python didn't have better support for the Perl one-liner
style of programming.

~~~
Drdrdrq
It still seems odd to me. While these projects are nice, if the functionality
is not supported in the core system, it is rarely worth the additional trouble
(imho). Situation is similar to gitless - git's cli interface sucks, but it is
available on every dev machine.

------
dbieber
This is fantastic -- I will certainly turn to this instead of awk/sed in many
instances. I expect I'll use this in place of particularly tricky greps too.
Really nice that you can seamless interleave standard bash commands with pyp
commands, if e.g. an existing bash tool can get a piece of the job done more
simply.

I already turn to Python in place of Bash whenever any nontrivial amount of
complexity is needed or if any reuse is expected in the script I'm writing (I
do this with Python Fire). Now I think I'll be making that Bash->Python switch
at even lower levels of complexity, and even when no reuse is expected at all.

------
tehalex
Here's the ruby equivalent:

[https://github.com/thisredone/rb](https://github.com/thisredone/rb)

~~~
freedomben
Was hoping to find a comment just like yours! Thanks!

I use Ruby from the cli frequently for stuff that exceeds an easy jq
expression or if jq is not availabel but Ruby is (which happens a lot on RHEL
systems when EPEL is not enabled, which is the case in a lot of banks) Ruby's
awk -> perl -> ruby heritage endows it with a pretty good cli mode, but
sometimes it's verbose. For example parsing json:

    
    
        curl https://something \
          | ruby -r json -e "puts JSON.parse(STDIN.read)['data']['ip']"
    

But obviously that's wordier than it needs. (To pre-empt the "just use jq!"
people, I agree. For simple examples I would just use jq. This example here is
just an example).

------
dagmx
There exists a very similar tool for several years with the exact same name

[https://opensource.imageworks.com/pyp.html](https://opensource.imageworks.com/pyp.html)

~~~
hauntsaninja
Yup, that was the inspiration for this project and its name. See
[https://github.com/hauntsaninja/pyp#related-
projects](https://github.com/hauntsaninja/pyp#related-projects) for more
discussion. By no means is this a new idea, but I think this iteration has
some convenient twists :-)

~~~
nightcracker
Yanking another projects name is more than just 'inspiration'. I would
consider it a very hostile action.

~~~
hauntsaninja
Given that the original project has been unmaintained for several years and is
Python 2 only, I think it's not unreasonable. There's very little reason to
have both installed, so I don't think it's a problem for users either.

It's also just the perfect name! It's punny, it's short and it acknowledges
prior art. And of course, if the original author took issue with it, I would
change the name.

~~~
andybak
Not piling on but - did you pop him an email to check?

------
Evanbenn
Brilliant, just the right amount of magic. The explain feature is what really
sold me that you thought about users! I wonder if I can put a bit of
humanfriendly support in there with a pr.

------
daenz
Very cool! I'm the author of sh.py, and seeing projects that use just the
right amount of magic to accomplish something unique and powerful is
inspiring. Great work and good luck with pyp!

~~~
divbzero
[sh.py]: [https://amoffat.github.io/sh/](https://amoffat.github.io/sh/)

------
akubera
Also related is pythonpy[0], which added a similarly named command 'py', but
apparently that project has died.

This looks like a big improvement, with math functions auto-imported, and the
--script option is a fantastic idea.

Good job.

[0.0] [https://pypi.org/project/pythonpy](https://pypi.org/project/pythonpy)

[0.1]
[https://news.ycombinator.com/item?id=8158976](https://news.ycombinator.com/item?id=8158976)

------
vertak
This is very cool. This is faster than postman for me if I was to use
something like you showed in the jq example for quickly playing around with a
public api’s json response.

------
slightwinder
Something like this should be baked straight into CPython itself. There are so
many similar projects, there clearly is some demand for this. This could boost
pythons usage for shell-jobs significantly if people have a reliable
installation of this behaviour.

------
kortex
This is going straight into the toolkit. The main thing that keeps me from
doing more inline python is all of the printing boilerplate.

Well, two things, the second thing being performance.

How's the startup time? I find it hard to get python utils under a few hundred
ms just to spin up.

~~~
hauntsaninja
Thanks for checking pyp out!

Startup time hasn't been particularly perceptible in my use of it. Just ran a
quick benchmark on my (old, not powerful) laptop:

    
    
      ~ λ hyperfine 'pyp x'
      Benchmark #1: pyp x
        Time (mean ± σ):      75.9 ms ±   1.2 ms    [User: 55.9 ms, System: 15.1 ms]
        Range (min … max):    74.5 ms …  79.0 ms    36 runs
    

Since there isn't any input, this should just measure startup and AST
transformation time.

------
rasengan
I saw this and just made this real quick - a JS equivalent.

[https://github.com/realrasengan/jpipe](https://github.com/realrasengan/jpipe)

~~~
eloahx
Thanks! This will make things a little easier for me as I move in to
sysadminning and JS development.

Surprised it didnt exist already

------
hauntsaninja
Thanks to everyone who checked pyp out!

Not sure if this comment will ever get seen, but I just added the ability to
configure pyp with statically analysed Python, so it's easy to define your own
functions, import aliases, or use libraries like pipetools to make your life
easier! More details at [https://github.com/hauntsaninja/pyp#pyp-is-
configurable](https://github.com/hauntsaninja/pyp#pyp-is-configurable)

------
gigatexal
Ok, I've been looking for something like this. Very cool indeed!

------
beamatronic
A lot of these projects remind me of the Smalltalk image concept - I used to
develop on Digitalk Smalltalk/V on Windows 3.1 in the mid 90s.

------
isignal
Pretty cool!

How would one accomplish the following with pyp?

* Wait until one sees a line with a start marker (say 'start:')

* Perform on an operation on the lines until an end marker ('end:') is reached.

Would you do a whole-input operation on stdin to extract the relevant lines
first and then operate on the individual lines? Or would you maintain boolean
conditionals within the loop?

~~~
hauntsaninja
There are a number of possible options! And remember you can always pipe pyp
to pyp. Here are some ideas, in order of what came to mind, where the
operation is incrementing.

    
    
      pyp 'lines[lines.index("start") + 1:]' | pyp 'lines[:lines.index("end")]' | pyp 'int(x) + 1'
      
      pyp 'start = lines.index("start"); end = lines.index("end", start); map(lambda x: int(x) + 1, lines[start + 1:end])'
      
      pyp 'dropwhile(lambda x: x != "start", lines)' | pyp 'lines[1:]' | pyp 'takewhile(lambda x: x != "end", lines)' | pyp 'int(x) + 1'
      
      pyp 'map(lambda x: int(x) + 1, takewhile(lambda x: x != "end", islice(dropwhile(lambda x: x != "start", lines), 1, None)))'
      
      pyp -b 'cond = False' 'if cond and x == "end": break' 'if cond: print(int(x) + 1)' 'if x == "start": cond = True'

------
beagle3
Ubercool. There's a similar project called "pythonpy" which I had used in the
past.

It seems to have disappeared off the face of the earth, the only thing I found
is this fork: [https://github.com/fish2000/pythonpy-
fork](https://github.com/fish2000/pythonpy-fork)

------
Hello71
I was going to ask "isn't this basically just awk" but the README quickly
cleared that up! Excellent documentation.

One thing I'm wondering is how's the startup time/performance in general? awk
is kinda slow overall, but Python starts a little too slow to be used in shell
scripting.

~~~
Hello71
Looks like there are no dependencies, so it should start pretty fast.

And sure enough, it's both easy and reasonably fast to run:

    
    
      $ time seq 1 10000 | awk '{a+=$0}END{print a}'
      50005000
      seq 1 10000  0.00s user 0.00s system 91% cpu 0.002 total
      awk '{a+=$0}END{print a}'  0.01s user 0.00s system 95% cpu 0.008 total
      $ time seq 1 10000 | python3 pyp.py 'sum(map(int, lines))'
      50005000
      seq 1 10000  0.00s user 0.00s system 80% cpu 0.002 total
      python3 pyp.py 'sum(map(int, lines))'  0.06s user 0.01s system 99% cpu 0.070 total
    

I wouldn't run it in a loop, but it's definitely a contender even in high-ish
performance situations.

------
emblaegh
I remember using this waay back in the day when this was still hosted on
google code, and having a really good experience with it, despite some
performance issues. I thought the project was dead, but I'm glad it's still
going on.

------
geophile
Along similar lines:
[https://github.com/geophile/marcel](https://github.com/geophile/marcel)

------
michaelmior
An older attempt at this is piep

[https://github.com/timbertson/piep](https://github.com/timbertson/piep)

------
auscompgeek
Yet another similar project:
[https://github.com/edk0/spy](https://github.com/edk0/spy)

