Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Pyp – Easily, magically, run Python at the shell (github.com/hauntsaninja)
285 points by hauntsaninja on May 9, 2020 | hide | past | favorite | 42 comments



Mario is a very featureful application of similar ideas, and includes multiple built-in transformations, concurrency support, user-defined commands, and extensive documentation.

https://github.com/python-mario/mario


Thanks for sharing! I hadn't seen Mario before. The async support is super nifty! From a quick look, it looks like pyp might be a little quicker for some use cases since it doesn't need subcommands, but I'll definitely be updating https://github.com/hauntsaninja/pyp#related-projects


+1 for mario.

It even has addons which enables extras like:

https://mario-addons.readthedocs.io/en/latest/cli_reference....


My cousin helped create xonsh (https://xon.sh), a POSIX-compliant "python as drop-in shell replacement", years ago.


xonsh is great! I'm a fan of your cousin's work :-) I mentioned it in https://github.com/hauntsaninja/pyp#related-projects for if you want even more Python in your shell.


Cool! It's mentioned in pyp's README:

"xonsh is a shell whose language is a superset of Python; this is more ambitious and pretty different from pyp. pyp is easier to use for the one-liner piping use case, but if you need more Python in your shell, check out xonsh."


Thanks! Xonsh is my daily driver. With the vox plugin, don't even need virtualenv


Nice. I made something similar, but for JavaScript https://github.com/jeswin/basho

It's very useful to be able to automate with a language you already know. I use it quite a bit now, for instance to figure out which of my git repos need a push or a pull.


This made me nostalgic for a utility I wrote back in 2005 called `pyline`:

http://code.activestate.com/recipes/437932-pyline-a-grep-lik...

I'm sure I wasn't the first to scratch that particular itch. It always seemed odd, back then, that Python didn't have better support for the Perl one-liner style of programming.


It still seems odd to me. While these projects are nice, if the functionality is not supported in the core system, it is rarely worth the additional trouble (imho). Situation is similar to gitless - git's cli interface sucks, but it is available on every dev machine.


This is fantastic -- I will certainly turn to this instead of awk/sed in many instances. I expect I'll use this in place of particularly tricky greps too. Really nice that you can seamless interleave standard bash commands with pyp commands, if e.g. an existing bash tool can get a piece of the job done more simply.

I already turn to Python in place of Bash whenever any nontrivial amount of complexity is needed or if any reuse is expected in the script I'm writing (I do this with Python Fire). Now I think I'll be making that Bash->Python switch at even lower levels of complexity, and even when no reuse is expected at all.


Here's the ruby equivalent:

https://github.com/thisredone/rb


Was hoping to find a comment just like yours! Thanks!

I use Ruby from the cli frequently for stuff that exceeds an easy jq expression or if jq is not availabel but Ruby is (which happens a lot on RHEL systems when EPEL is not enabled, which is the case in a lot of banks) Ruby's awk -> perl -> ruby heritage endows it with a pretty good cli mode, but sometimes it's verbose. For example parsing json:

    curl https://something \
      | ruby -r json -e "puts JSON.parse(STDIN.read)['data']['ip']"
But obviously that's wordier than it needs. (To pre-empt the "just use jq!" people, I agree. For simple examples I would just use jq. This example here is just an example).


Awesome! Thanks!


There exists a very similar tool for several years with the exact same name

https://opensource.imageworks.com/pyp.html


Yup, that was the inspiration for this project and its name. See https://github.com/hauntsaninja/pyp#related-projects for more discussion. By no means is this a new idea, but I think this iteration has some convenient twists :-)


Yanking another projects name is more than just 'inspiration'. I would consider it a very hostile action.


Given that the original project has been unmaintained for several years and is Python 2 only, I think it's not unreasonable. There's very little reason to have both installed, so I don't think it's a problem for users either.

It's also just the perfect name! It's punny, it's short and it acknowledges prior art. And of course, if the original author took issue with it, I would change the name.


Not piling on but - did you pop him an email to check?


Brilliant, just the right amount of magic. The explain feature is what really sold me that you thought about users! I wonder if I can put a bit of humanfriendly support in there with a pr.


Very cool! I'm the author of sh.py, and seeing projects that use just the right amount of magic to accomplish something unique and powerful is inspiring. Great work and good luck with pyp!


[sh.py]: https://amoffat.github.io/sh/


Also related is pythonpy[0], which added a similarly named command 'py', but apparently that project has died.

This looks like a big improvement, with math functions auto-imported, and the --script option is a fantastic idea.

Good job.

[0.0] https://pypi.org/project/pythonpy

[0.1] https://news.ycombinator.com/item?id=8158976


This is very cool. This is faster than postman for me if I was to use something like you showed in the jq example for quickly playing around with a public api’s json response.


Something like this should be baked straight into CPython itself. There are so many similar projects, there clearly is some demand for this. This could boost pythons usage for shell-jobs significantly if people have a reliable installation of this behaviour.


This is going straight into the toolkit. The main thing that keeps me from doing more inline python is all of the printing boilerplate.

Well, two things, the second thing being performance.

How's the startup time? I find it hard to get python utils under a few hundred ms just to spin up.


Thanks for checking pyp out!

Startup time hasn't been particularly perceptible in my use of it. Just ran a quick benchmark on my (old, not powerful) laptop:

  ~ λ hyperfine 'pyp x'
  Benchmark #1: pyp x
    Time (mean ± σ):      75.9 ms ±   1.2 ms    [User: 55.9 ms, System: 15.1 ms]
    Range (min … max):    74.5 ms …  79.0 ms    36 runs
Since there isn't any input, this should just measure startup and AST transformation time.


I saw this and just made this real quick - a JS equivalent.

https://github.com/realrasengan/jpipe


Thanks! This will make things a little easier for me as I move in to sysadminning and JS development.

Surprised it didnt exist already


Thanks to everyone who checked pyp out!

Not sure if this comment will ever get seen, but I just added the ability to configure pyp with statically analysed Python, so it's easy to define your own functions, import aliases, or use libraries like pipetools to make your life easier! More details at https://github.com/hauntsaninja/pyp#pyp-is-configurable


Ok, I've been looking for something like this. Very cool indeed!


A lot of these projects remind me of the Smalltalk image concept - I used to develop on Digitalk Smalltalk/V on Windows 3.1 in the mid 90s.


Pretty cool!

How would one accomplish the following with pyp?

* Wait until one sees a line with a start marker (say 'start:')

* Perform on an operation on the lines until an end marker ('end:') is reached.

Would you do a whole-input operation on stdin to extract the relevant lines first and then operate on the individual lines? Or would you maintain boolean conditionals within the loop?


There are a number of possible options! And remember you can always pipe pyp to pyp. Here are some ideas, in order of what came to mind, where the operation is incrementing.

  pyp 'lines[lines.index("start") + 1:]' | pyp 'lines[:lines.index("end")]' | pyp 'int(x) + 1'
  
  pyp 'start = lines.index("start"); end = lines.index("end", start); map(lambda x: int(x) + 1, lines[start + 1:end])'
  
  pyp 'dropwhile(lambda x: x != "start", lines)' | pyp 'lines[1:]' | pyp 'takewhile(lambda x: x != "end", lines)' | pyp 'int(x) + 1'
  
  pyp 'map(lambda x: int(x) + 1, takewhile(lambda x: x != "end", islice(dropwhile(lambda x: x != "start", lines), 1, None)))'
  
  pyp -b 'cond = False' 'if cond and x == "end": break' 'if cond: print(int(x) + 1)' 'if x == "start": cond = True'


Having moved from Perl to Python many years ago, this is one of those things that Perl has magic to handle, for example:

    perl -lne '$sum += $_ if /start:/ .. /end:/; END{print $sum}' file.txt
With the -n option, perl runs your code for every line in the input (stdin, or filenames). $_ holds the content of the line. And the .. operator is a stateful (!) operator that becomes true when its left part becomes true, until the right part becomes true.

So I write 95% of my code in Python, but for command line magic, it's hard to beat Perl.


Ubercool. There's a similar project called "pythonpy" which I had used in the past.

It seems to have disappeared off the face of the earth, the only thing I found is this fork: https://github.com/fish2000/pythonpy-fork


I was going to ask "isn't this basically just awk" but the README quickly cleared that up! Excellent documentation.

One thing I'm wondering is how's the startup time/performance in general? awk is kinda slow overall, but Python starts a little too slow to be used in shell scripting.


Looks like there are no dependencies, so it should start pretty fast.

And sure enough, it's both easy and reasonably fast to run:

  $ time seq 1 10000 | awk '{a+=$0}END{print a}'
  50005000
  seq 1 10000  0.00s user 0.00s system 91% cpu 0.002 total
  awk '{a+=$0}END{print a}'  0.01s user 0.00s system 95% cpu 0.008 total
  $ time seq 1 10000 | python3 pyp.py 'sum(map(int, lines))'
  50005000
  seq 1 10000  0.00s user 0.00s system 80% cpu 0.002 total
  python3 pyp.py 'sum(map(int, lines))'  0.06s user 0.01s system 99% cpu 0.070 total
I wouldn't run it in a loop, but it's definitely a contender even in high-ish performance situations.


I remember using this waay back in the day when this was still hosted on google code, and having a really good experience with it, despite some performance issues. I thought the project was dead, but I'm glad it's still going on.


Along similar lines: https://github.com/geophile/marcel


An older attempt at this is piep

https://github.com/timbertson/piep


Yet another similar project: https://github.com/edk0/spy




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: