
Show HN: Bsed – Simple, English syntax on top of Perl text processing - bihla
https://github.com/andrewbihl/bsed
======
peteretep
Kinda off that this is written in Python, and not Perl. If it was written in
Perl, it'd have the portability that the underlying translated commands have!

I'd be happy to help you port it to modern, high-quality, readable Perl if
that's something you're interested in. Being able to install this by dropping
a single Perl file into ~/bin/ would be neat.

~~~
bihla
I agree, if it were entirely Perl it would have essentially no dependencies.
Right now the audience is limited to those with a currently functioning
Python3 environment or willing to configure one.

My only concerns are:

I do not know what the process of getting comparable tab completion would be
in Perl. I depended on the argcomplete package for this feature which I
consider to be essential...the clarity of the syntax depends on being a bit
verbose, and this is only usable because tab completion makes it fast to use.

Also, I'm somewhat reluctant to develop in Perl, though I'm willing to do it
provided the benefit + demand is there. It would be ironic to have learned
Perl as a consequence of building a tool whose primary function is to prevent
one having to learn Perl.

I'll send you an email to discuss further.

~~~
acqq
I haven't used the following (I simply never had to implement the shell
autocompletion as try I avoid the shell as much as I can, preferring to
organize my code in the actual files that I can reuse from my editor) but I
guess it can be the best start to include significant parts of this code
inside of your final script, if the goal can be to have a single easily usable
script doing everything, as somebody mentioned that ideal here (still take
care to keep the compatible/same license):

[https://fastapi.metacpan.org/source/PERLANCAR/Complete-
Bash-...](https://fastapi.metacpan.org/source/PERLANCAR/Complete-
Bash-0.320/lib/Complete/Bash.pm)

You can surely omit the lines between =pod and =cur (they are used to create
the documentation) and most probably also the "description" parts.

> It would be ironic to have learned Perl as a consequence of building a tool
> whose primary function is to prevent one having to learn Perl.

It's less ironic than you think, I consider it the bigger irony that Python
turned to be the major compatibility problem in all but single-user single-
program environments (it's really not trivial to have to work with more Python
variants as the dependencies for different code). You can be sure that any
Perl 5 skills you obtain can remain quite portable and usable across the
different environments.

I've also already written here, I consider Perl significantly "safer" language
than Python: when I modify some big Perl scripts, very often the Perl compiler
(when used with the "use strict" and -w option) will tell me early enough what
is missing for a program to function correctly again, as Perl simply more
explicit, in some weird way almost like the typed languages. I never have the
same feeling of "it will work" with the modification of the Python scripts.

That "line noise" is actually meaningful, much more consistent than the shell
languages, and safer in case of the changes than Python.

------
ilovetux
Perl aside, I would say that's almost imperative to learn sed and awk. They
invaluable tools for getting things done in *nix. They seem cryptic at first,
but put a weekend into learning them and you learn that they are simply
concise.

~~~
wink
And then you have a script that needs to run on a mac as well and you can
either fight that the people will install the gnu version or you just rewrite
your one-liner in perl.

I'm not a huge fan of perl but I've been writing my one-liners in it for a few
years now and I've had less problems with portability.

~~~
ilovetux
I would use another language for distributing a script (probably) python. Awk
and sed I use more on a one-off basis although I do have a list of ones I use
frequently.

~~~
masukomi
Would that be the deprecated 2.x with OS X or 3.x which has been available for
years?

To put it another way, you've replaced sed incompatibility with python
incompatibility. It's not really a win.

I'm not saying Perl is the best choice here, but it's a better choice than
Python for cross platform compatibility.

------
disposedtrolley
This is cool! I’ve wanted to explore something similar but incorporating a
broad range of Unix commands.

I imagined a drag and drop interface (sort of like the Scratch programming
language) with English descriptors of the functions being performed. You’d
make it a web UI or something that generates a bash script you can paste into
the terminal.

------
asicsp
some notes on the sample awk/sed/perl one-liner given:

    
    
        # input line has to be explicitly printed
        awk '{gsub(/Jack/,"Jill")} 1' file.txt
    
        # -i will do inplace editing, unlike the awk command
        # -i by itself won't work on non-GNU versions, needs backup extension
        sed 's/Jack/Jill/g' file.txt
    
        # use single quotes always, unless double is needed
        # -p will behave like default sed
        perl -pe 's/Jack/Jill/g' file.txt
    

personally, I prefer the terseness of these commands over verbose SQL like
syntax (and also the fact that I don't know SQL like tools)

However, I would agree that initial learning curve is tough for sed/awk/perl.
Once you get familiar with their idioms, they become the swiss army knife of
cli text processing (along with grep). I have an entire repo dedicated to such
tools[1]

[1] [https://github.com/learnbyexample/Command-line-text-
processi...](https://github.com/learnbyexample/Command-line-text-processing)

~~~
chmln
Shameless plug for even terser syntax:
[https://github.com/chmln/sd](https://github.com/chmln/sd)

~~~
bloopernova
That looks very nice, thank you for sharing/writing it!

Are you responsible for updating the Cargo package? It's the first time I've
used Cargo and was wondering how often I should re-run it or check for
updates?

~~~
chmln
You can run cargo install-update -a every now and then to check for updates.
At some point in a few weeks I hope to have binaries distributed with every
release, which means there will be better/faster ways to install sd than
cargo.

------
bihla
I'd love to hear any questions or feedback. I've gotten quite a bit of use
from this tool personally and have had a lot of fun making it.

~~~
king_magic
Really looking forward to trying this out. The syntax looks very clean &
clear. I hate the insanity of sed, and assuming the performance is here, I’d
switch to this in a heartbeat.

~~~
bihla
In my experience, Perl outperforms sed on most equivalent commands. I should
do some benchmarking and add it to the documentation.

~~~
rurban
You should really use psed then, which came with perl5 from 5.002 until
v5.21.1 and is now on CPAN. This is a proper sed, implemented in perl.

[https://metacpan.org/pod/release/RJBS/perl-5.21.0/x2p/s2p.PL](https://metacpan.org/pod/release/RJBS/perl-5.21.0/x2p/s2p.PL)
[https://metacpan.org/pod/release/LEONT/App-s2p-1.000/script/...](https://metacpan.org/pod/release/LEONT/App-s2p-1.000/script/s2p)

------
jaytaylor
I'd definitely incorporate it into my projects if it were ported Go, Rust, or
any reasonable redistributable form, and continued to use a BSD license. Way
clearer than cryptic sed and awk one-liners! Big kudos to you for making this
and sharing it with the world!

Python apps which are more than a single script are a non-starter for me. It's
a big pain to get everyone on my 120+ person dev team to install it properly,
they're all over the place with regard to environments. "Oops, I'm on Python
3.4", or python 2.6, or "I use this other python path"; it's inevitably a
headache.

~~~
yoklov
Yep, this. Python is a headache for this sort of thing IME. (Nothing against
the language though, although I don't know it)

~~~
anoncake
> IME

> I don't know it

Huh?

~~~
elliotec
One can have “experience” with something, and not “know” it.

Could just as easy say: “I have experience in scenarios where I must use
python or run its scripts in an evironment, and it has been a headache, but I
can’t say I dislike the language because I haven’t needed to explicitly create
programswith it.”

~~~
yoklov
This is exactly what I meant to convey, thanks.

------
Sir_Cmpwn
This is very cool for simplifying interactive use :) but I'd like to caution
people away from immortalizing it in their scripts. sed and awk are both
standardized by POSIX and can be found on pretty much any system, and
trivially ported to new systems. Porting Python, on the other hand, is
comparitavely a mammoth proposition, and often impossible on some platforms.
For this reason, I'm generally allergic to "new $x replaces $y!" for any $y
which is governed by portable standards.

~~~
bihla
I agree. For anything longer term, I'd recommend finding the right bsed
command and then using -t flag to get the Perl translation.

Perl is installed by default on common operating systems.

~~~
Sir_Cmpwn
>Perl is installed by default on common operating systems.

Aye, but what about the uncommon ones? For your consideration, I'd like to
share Perl's configure script:

[https://github.com/Perl/perl5/blob/blead/Configure](https://github.com/Perl/perl5/blob/blead/Configure)

It's my favorite configure script of all time, but I would hate to be the guy
who has to extend it with a new platform :) And I have!

~~~
username223
This is something I love about Perl: in an age of many subtly incompatible
operating systems, Perl said "F it, we'll do what it takes to compile and run
your program everywhere." People don't appreciate this now, when everything is
Linux, Mac, or Windows, but back when you regularly had to deal with various
BSDs, HPUX, AIX, Mac OS 9, and even the occasional VAX, Perl was a godsend.

~~~
hpcjoe
$_ =~ s/was a/is a/g;

There, I fixed it for you.

Seriously, I see so many ways of re-inventing these wheels, written in the fad
languages/systems d'jour. Perl works very well, for a large subset of tasks
that are common.

The same code that ran 25+ years ago (in perl4) on my Sun and Irix machines,
runs today on my laptops in linux, MacOSX. I remember building Perl atop the
Cray J90 running Unicos in grad school, as I had built run automation software
in it (not a queuing system, but code to encapsulate some of my runs and
return the relevant data). I had developed that code on my OS2 based desktop
with Perl.

Perl is a go-to language. It gets stuff done. Without pain.

------
jancsika
> bsed giant_malformatted.json replace '\'' with '\"' | bsed replace 'True'
> with 'true' | bsed replace 'False' with 'false'

bsed giant_malformatted.json replace '\'' _that begins or ends a string_ with
'\"' | bsed replace _unquoted_ 'True' with 'true' | bsed replace _unquoted_
'False' with 'false'

~~~
bihla
Fair point. I mostly put those commands together to illustrate concepts. Two
things you've touched on though that I haven't implemented yet: compound
conditionals, and a nice way to indicated word boundaries (as opposed to the
regex solution)

------
mhd

        TEXT PROCESSING DIVISION.
        PERFORM REGEXP-REPLACE "LETTER A; LETTER B; ONE-OR-MORE-CHARACTERS;" EMPTY-STRING
        .

~~~
JdeBP
Or, rather:

    
    
        PIC A(2)X(integer01)

------
seiferteric
Neat, but a little scary because you rely on it to interpret your intentions
correctly. Usually when I am trying to do something, I build up my command in
multiple steps, printing outputs and verifying it is correct before actually
running the command. Maybe you could have a "diff" or "dryrun" mode that would
show the potential changes. Another idea would be for this script to "compile"
a CLI command using grep/awk/sed etc instead of doing the actual editing
itself.

~~~
bihla
The interpretation is only translating into standard Perl one-liners, and by
default the execution writes to stdout instead of modifying the file. I
haven't written code to actually do the text transformations.

My normal workflow is to write commands, pipe them together as needed, and
when I think I am satisfied I'll store to a temporary file. If I really need
to be extra sure then I will use my preferred ddiff tool to compare against
the original before doing the swap.

I may have mis-interpreted but I hope between this and the unit tests your
concerns are alleviated

~~~
seiferteric
Okay I misunderstood, thanks!

------
kylek
Great, now I only have to google SQL syntax :)

(Sorry, had to take the easy one. Tooling using something as wide-spread as
SQL is definitely a good thing!)

~~~
collyw
Its worth your time learning. Its been an in demand skill for decades now.
Compare that to JavaScript framemworks....

------
chmln
How does it compare performance-wise to other tools with similar goals? Seeing
this is written in 100% python, I suspect it will be an order of magnitude
slower.

~~~
bihla
The python code only carries out translation into Perl. The actual execution
of the text transformation is Perl, which is generally the fastest of the
available tools.

------
HerrMonnezza
The HN entry title might be misleading: this is a _Python_ tool, Perl is only
mentioned in the Readme for comparison.

~~~
bihla
Perl is actually executing the text transformations. The python is a
translation layer between the bsed syntax and the Perl command, so you get the
performance of Perl with a very minor time cost for interpretation done in
Python.

~~~
singingfish
Yeah, there's not a lot of point in having the python layer. Perl excels at
these kinds of things. I'm guessing that the original developer just doesn't
know perl well enough to implement that part. Which is not to denegrate his
work, but this could easily be a drop in single file perl executable if
someone does the actual (not that much) work.

------
purplezooey
Ah, googling for some awk wizardry. Best done with a sandwich.

------
salutonmundo
Odd that it installs with pip but uses perl.

------
PhantomBKB
Based

------
ncmncm
He lost me at "Perl".

~~~
bihla
Can you elaborate? The tool doesn't require you to learn/know perl, that's
just what operates under the hood. I actually chose to exclude any reference
to Perl in the title, though it appears a moderator changed it.

~~~
otherflavors
they are trying for cool points by hating on perl

~~~
Yetanfou
...which is such an odd thing to do on a forum frequented by people who pride
themselves on their breadth of knowledge and openness to new ideas.

