
How to Make Money Using Grep, Sed and Awk - bsdpunk
https://openmonstervision.github.io/blog/posts/how-to-make-money-using-grep-sed-and-awk/
======
nrki
"How to make money by posting a pretty boring article to HN and asking for
book purchases at the end of it"

~~~
janvdberg
I didn't mind, I actually lolled when reading his other blogpost about how he
botched the personal question part of an interview:
[https://openmonstervision.github.io/blog/posts/how-not-to-
ge...](https://openmonstervision.github.io/blog/posts/how-not-to-get-a-job-
with-an-elite-tech-company/)

~~~
nrki
That article /is/ funny.

It also doesn't beg for book purchases for books he didn't write... :P

------
mar77i
Dude! You need help with using your tools' features to reduce your pipeline
lengths.

\- grep derp | sed whatever is the same as sed '/derp/ whatever' (you might be
interested in sed -r for that matter)

\- sed whatever | sed whatever\ else is the same as sed -e whatever;whatever\
else

This list is by no means complete but saves a lot of external processes
already, and seriously: think about writing the whole thing in pure bash or
awk. There might be just too little gain to justify including all these tools
without composing the many features they provide.

~~~
CJefferson
I disagree, unless it's slow enough to effect performance, I prefer to combine
simple pipes.

The only time I have to use more complicated constructions is to get around
the stupid problem that passing filenames with pipes is almost impossible to
do safely.

~~~
tslug
I learned when writing bash scripts for cygwin under Windows that Windows
doesn't support copy-on-write fork()'s. This means that any new process is
extremely expensive to fork, because it copies the memory from the process it
was forking from, even if it doesn't need it.

As a result, I did as much as I could using only bash internals, and very
rarely did I use pipes, because they always fork()'d a bash subshell in
addition to whatever the process was.

In about 7k lines of bash script written to completely automate the iterative
development of two games with a shared game engine featuring dockerized server
containers, I was able to avoid using sed in almost all of it. When I did use
it (pretty sure only two places), it was basically for mass substitution of
variables inside of text files, and the multi-line syntax worked very nicely
for this, as I could form the sed line with a loop over all the variables I
wanted to replace.

It turns out for most bash scripts, you can do the most common sed
substitutions using bash's rich variable substitution expressions. For
instance, you see in a lot of bash scripts calling commands like "dirname" and
"basename" to get the directory and filename of paths. There's a much faster
way to do this in bash:

    
    
      path=/tmp/my/path/to/stuff.txt
      dir="${path%/*}"
      file="${path##*/}"
      other="${path/stuff/other}"
    

That dir line means "delete the shortest string that matches /* from the end
of the string". That file line means "delete the longest string matching */
from the start of the string." That other line means "replace the word stuff
with the word other in this string".

------
yomansat
I looked up the freelance job in question, it pays $36 for a 215 page
publication from the 80s, including checking where automation has failed.

A question to those who do freelance work: sites like Upwork often have low
budgets, where do you guys find better projects with better compensation?

~~~
iDemonix
I gave up with all of them. A friend of mine is using Upwork and Freelancer
(awful) to find gigs, and he tried to get me involved so I could help him out
with overspill. It seemed like you had to put ludicrously low rates to get
terrible work, and despite paying peanuts, everyone expects amazing results.

I ditched it, joined some communities online, did a tiny bit of networking,
and announced I was looking for work related to X - that was far more fruitful
than sending 10 pitches a day for gigs on those sites and getting nothing in
response because someone in India will do it for a dollar. Although I did love
seeing the same projects re-appear a week later to fix the crap that their
offshore team delivered.

~~~
xrisk
I'm from India and I'm horrified that so many people are willing to do work
for nothing or next to nothing...

------
tragomaskhalos
A man who eschews Perl, and all later and more usable tools, in favour of a
bastard hodgepodge of shell and awk with a smattering of sed: It's like
looking in the mirror.

------
angarg12
At what point it pays off to switch to python/node instead of keep growing a
bash script?

~~~
Xophmeister
The beauty of bash scripts is that anything with u+x in your PATH, aliases and
user-defined functions become a first-class citizen with the same interface.
The usual plumbing with pipes and redirection -- and sometimes some more
advanced stuff like process substitution -- is often all you need.

Moving to Python, say, makes the control flow and the "software engineering"
easier, for sure. However, don't underestimate the power of grep, sed and
especially awk. I wouldn't want to reimplement a half-arsed, presumably non-
bug-free version of awk in Python when I could have just used awk in the first
place.

~~~
tinix
There's no reason you can't orchestrate awk and other tools from python
though. (I doubt I'm the only one that's done that before).

~~~
zwischenzug
I wrote a tool to do that:

[https://github.com/ianmiell/shutit](https://github.com/ianmiell/shutit)

Among many other things I use it to test Kubernetes/OpenShift clusters in
Vagrant for Chef recipes:

[https://github.com/ianmiell/shutit-openshift-
cluster/blob/ma...](https://github.com/ianmiell/shutit-openshift-
cluster/blob/master/shutit_openshift_cluster.py)

and here's some others in a more 'native' python format:

[https://github.com/ianmiell/shutit-
scripts/](https://github.com/ianmiell/shutit-scripts/)

[https://github.com/ianmiell/shutit-
scripts/blob/master/logme...](https://github.com/ianmiell/shutit-
scripts/blob/master/logmein/logmein.py)

[https://github.com/ianmiell/shutit-
scripts/blob/master/gnupl...](https://github.com/ianmiell/shutit-
scripts/blob/master/gnuplot/graphs.py)

~~~
KGIII
> crazy talk about pushing awk through python

> oh! I wrote a tool to do that!

 _sniffles_

That might just be my favorite HN exchange ever. Y'all remind me of emacs
users. Between this and the unusual number of Perl mentions on HN today, my
cockles are suitably heated.

That _is_ a compliment. Nary a day passes where HN doesn't give me a reason to
keep clicking links and learning.

I started off giggling and now I'm moved from my tablet to a laptop so that I
can investigate your shutit - as it looks like a handy tool for learning. The
scales look the most interesting.

I have been here for a while, but only recently (past couple of months)
decided to comment. I lurked for like 12 months, just to see if I'd fit in.
Why? HN continually has commentary about things I haven't yet learned.

In short, this is my awkward attempt to thank you. I'll be spending the
afternoon trying to enjoy your shutit scales.

~~~
zwischenzug
Great, thanks!

You can find more info here on my blog:

[https://zwischenzugs.wordpress.com/](https://zwischenzugs.wordpress.com/)

do ping me with your experience.

------
ggm
The best way to make money with grep/sed/awk is to become sufficiently l33t in
them, you can be gainfully employed by others to "do things" on "the goddam
unixes"

hey.. we got more unixes here. can somebody grep them out the way?

~~~
zero_one_one
grep -v .*[nN].x will solve the infrastructure problem...

As you say, the way to make money (with any tool) is to get good enough at
using them to be useful to someone that wants to pay you to do so

------
nunez
If author used better variable names, then the code would not have been as
hard to look at

~~~
bsdpunk
That's fair

