
Adventures in /usr/bin and the likes - mubaris
http://ablagoev.github.io/linux/adventures/commands/2017/02/19/adventures-in-usr-bin.html
======
camtarn
Worth it just for learning that there's a "replace" command, for multi-file
search-and-replace without having to remember sed syntax.

Can't believe I've been running Linux for 16 years and I didn't know about
that one.

~~~
kevhito
I was tempted to poke a little fun at you for having trouble with sed -- among
my peers, 's/foo/bar/g' used pretty much daily, even just in informal
emails/conversations.

But I'm glad I looked at the list first, because I found this wonder: `lstopo
--of txt`. I'm going to use this in class next week, I can't believe I've
never seen that before.

~~~
linkregister
My biggest hardship with sed is having to figure out how to escape the
operators within regular expressions on the command line. I think it's pretty
reasonable to spend 15+ minutes on a complicated replacement regex.

Edit: It looks like I agree with you; `replace` only solves trivial problems
that can easily be done with `sed`.

~~~
userbinator
You can use single quotes to escape everything except single quotes from the
shell. Then use backslashes to escape single quotes where needed.

If you haven't taken the time, I'd say it's well worth learning exactly how
the shell escapes work. It's surprisingly simple and natural once you get used
to it.

~~~
linkregister
Thanks for the advice; I should block some time to fully grok shell escapes.

Even with single quotes, sed still requires escaping parentheses, + operators,
and other operators that would otherwise be interpreted literally (but not all
operators should be escaped). In the languages that I learned regexes this
wasn't required.

~~~
NathanOsullivan
There's a command line switch -E for sed that gives you what we think of as
"normal" regex escaping.

~~~
linkregister
You have saved me many hours of future command line hacking. Thanks!

------
noisy_boy
Reminds me of my sysadmin days when I was trying to learn shell scripting and
would lookup man pages of random binaries in /usr/bin. Before we could bring
down a box for nightly maintenance/backup, we would send a shutdown broadcast
notice like:

banner "Shutdown"|wall; banner "in 5 mins"|wall

I found out "write" command and setup a hacky little one-to-one chat script -
it was terrible and geeky but I really liked it.

I'm probably forgetting some more obscure commands that I enjoyed :)

PS: forgot to mention that when I told my boss about the "write" based
scripts, he gloated that he had setup a quiz program which supported multiple
participants using named pipes. Awesome guy :)

~~~
netcraft
banner is awesome - any way to make it output slowly instead of always at
once?

~~~
dous
while IFS= read -r line; do echo "$line"; sleep 0.1 ; done < <(banner Hello)

------
inopinatus
ISTR it was Æleen Frisch (author of _Essential System Administration_ ) who
recommended that all _nix admins take one day a year to read through the
entire manual sections (1) and (8), which I implemented thus:

$ cd /usr/bin ; man _

(and repeat for other bindirs)

You will be amazed at what you've forgotten, and what your system can do.

~~~
l0b0

      $ cd /usr/bin ; man
      What manual page do you want?
    

Is there a command to get all the man pages? I tried `man *` and the internal
help, but no luck.

Edit: q then Return did it. D'oh. The obvious follow-up question is how do I
go back to the _previous_ man page?

~~~
felipebueno
$ man man ;)

~~~
l0b0
Actually, I could not find anything relevant by searching it for "file" and
"next". I guess it's purely the responsibility of the pager, which presumably
is not part of man itself.

------
gravypod
After reading this I'm reminded that if an IDE just supported replacing
highlighted text after it was piped through a command it would have a lot
going for it. A lot of these are features some that IDEs have been lacking for
years yet they've been hiding away in /bin.

~~~
black_knight
Acme, the text editor from Plan 9 has this. Piping, inputing and outputing
from commands are some of the most basic operations there.

Check out:

\- [http://acme.cat-v.org/](http://acme.cat-v.org/)

\- Russ Cox' tour:
[https://www.youtube.com/watch?v=dP1xVpMPn8M](https://www.youtube.com/watch?v=dP1xVpMPn8M)

~~~
gravypod
Is there an updated version with less complicated mouse interfaces?

------
z1mm32m4n
What a list of gems!

Also, I noticed you can rewrite this function:

    
    
        # Copy an ip address from a domain
        # Usage cip domain.com
        function cip() {
            ip=$(resolveip -s $1)
            echo $ip | xsel -b
            echo $ip
        }
    

as a oneliner without needing process substitution or echo:

    
    
        function cip() {
            resolveip -s $1 | tee /dev/fd/2 | xsel -b
        }

~~~
akerl_
I'm never 100% sure I understand FD redirection, but I think you can shorten
that even further to:

    
    
        function cip() {
            resolveip -s $1 1>&2 | xsel -b
        }
    

Since this'll copy stdout to stderr, and then stdout will get eaten by xsel

~~~
tomsmeding
It won't _copy_ stdout to stderr; it'll _move_ stdout to stderr. Stdout will
be empty after the redirect, so no, thos won't work :)

~~~
akerl_
Have you tried it out? It worked on my system.

~~~
IsmaOlvey
The parent is correct, the given command redirects stdout to stderr, it's not
copied (so stdout will be empty).

~~~
akerl_
Again, have you tried it?

    
    
        # echo "foo" 1>&2 | sed 's/foo/bar/'
        foo
        bar
    

Edit: I've done some more testing, and discovered that the above works on zsh,
but not in bash

2nd Edit: Ahha! [http://www.cs.elte.hu/zsh-
manual/zsh_7.html](http://www.cs.elte.hu/zsh-manual/zsh_7.html) . So this is
because zsh w/ the stock config (MULTIOS option enabled) will open as many
outputs as you give it. So it can both copy FD 1's contents to FD 2 and to the
pipe'd command.

------
jwilk
Apparently "resolveip" is part of... MySQL.

You may want to use "getent ahosts" instead:

    
    
      $ getent ahosts en.wikipedia.org
      91.198.174.192  STREAM en.wikipedia.org
      91.198.174.192  DGRAM  
      91.198.174.192  RAW

~~~
floatboth
That's an interesting command! (note: on FreeBSD, just "hosts", there's no
"ahosts")

I usually use drill(1) (from unbound, which is part of base on FreeBSD), or
dig(1) on Linux boxes that don't have unbound.

Also turns out that that replace command is also part of MySQL. LOL WTF
MYSQL?!

------
cel1ne
A command I often use for a nice overview of running stuff:

    
    
      $ netstat --numeric-ports -elp46 | sort
    
        Active Internet connections (only servers)
        Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
        tcp        0      0 0.0.0.0:22223           0.0.0.0:*               LISTEN      root       12159       3370/sshd       
        tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      root       781140      4940/nginx -g daemo
        tcp        0      0 0.0.0.0:4000            0.0.0.0:*               LISTEN      postgres   780857      4916/postgrest  
        tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      root       13126       3537/master     
        tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      postgres   12526       3406/postgres   
        tcp6       0      0 ::1:5432                :::*                    LISTEN      postgres   12525       3406/postgres   
        tcp6       0      0 :::22223                :::*                    LISTEN      root       12168       3370/sshd       
        udp        0      0 0.0.0.0:123             0.0.0.0:*                           root       12479       3400/ntpd       
        udp        0      0 127.0.0.1:123           0.0.0.0:*                           root       12485       3400/ntpd       
        udp6       0      0 ::1:123                 :::*                                root       12487       3400/ntpd       
        udp6       0      0 :::123                  :::*                                root       12480       3400/ntpd

~~~
digi_owl
Apparently netstat is one of the commands being depreciated on Linux, to be
replaced by ss.

------
pjungwir
Wow this list is tremendous! It's fun to still discover new commands after all
these years.

I actually used `taskset -c` just recently, when I was running lots of sidekiq
processes and wanted to make sure they all used different cores. It helped me
get full utilization out of the box, whereas before I would frequently see
some sidekiqs competing for a core while other cores sat idle.

~~~
rdtsc
Be careful with taskset and chrt, assigning realtime priority to processes and
pinning on them on CPUs might lead to un-expected behavior, priority inversion
or your system locking down (say if you pgrep some kernel tasks/threads in
there as well by accident, I've done before).

~~~
pjungwir
Thank you for the warning! I can see how that could happen with chrt. I can't
work out how it could happen with just taskset. Are you saying to be careful
when combining them? That would make sense to me.

Also I can't figure out what pgrap is. Do you mean pgrep? I think you must
mean something else, because I don't see what pgrep has to do with priority
inversion. But it shares a manpage with pkill. . . .

I don't do real-time work, but I've always been curious to know more about it.
And priority inversion seems like an interesting and challenging problem.

~~~
rdtsc
> Also I can't figure out what pgrap is. Do you mean pgrep?

Sorry meant pgrep.

> I don't see what pgrep has to do with priority inversion.

Grepping processes doesn't have anything to do with priority inversions and
locking your system. But grepping processes then setting them to realtime
priority sched_fifo 99 or could wreck havok

> I don't do real-time work, but I've always been curious to know more about
> it. And priority inversion seems like an interesting and challenging
> problem.

It can be fun. It is kind of a separate world on its own. Interestingly most
of the popular OS-es by default are tuned for throughput not low latency. So
configuring and getting everything just right is an interesting challenge
(sometimes involving applying kernel patches, although lately that might not
be necessary as some have been mainlined).

------
zoidb
A lot of really cool stuff here! _shameless plug_ I have a small side-project
site that goes through a lot of common activities on the command line
[https://cmdchallenge.com](https://cmdchallenge.com). Some great ideas that I
think I will incorporate.

------
shakna
Going off a comment on the article, using mainly awk to find your commands, I
expanded it a little bit, as most commands I use take a first argument that
changes how it works. (Aliasing this command might not be a bad idea.)

    
    
        history | awk '{print $1 "__-__" $2}' | sort | uniq -c | sort -rn | head -30 | awk '{gsub("__-__", " ", $2); $2} !max{max=$1;}{r="";i=s=100*$1/max;while(i-->0)r=r"#";printf "%50s %5d %s %s", $2, $1, r, "\n";}'
    

It produces a histogram of the 30 most frequent used commands.

Unsurprisingly for me, `git commit` was first. Kinda surprisingly, `apt-get`
was third.

Caveat: I convert spaces to the symbol sequence: __-__ and back. If your
command includes that symbol sequence, it'll break something.

------
minxomat
Pipes are not exclusive to *nix based systems. Windows e.g. has had them for
quite some time, dating back to at least DOS 7[0] from Win95.

[0] -
[http://www.lagmonster.org/docs/DOS7/pipes.html](http://www.lagmonster.org/docs/DOS7/pipes.html)

~~~
LukeShu
In early versions, DOS pipes weren't real pipes. It would run the first
command and save the output to a temporary file; then once that was finished,
run the second command with the file as input. I'm not sure when it gained
real pipes.

~~~
pjc50
Windows pipes appear to _still_ be second-class citizens. Look at this mess:
[http://stackoverflow.com/questions/34504970/non-blocking-
rea...](http://stackoverflow.com/questions/34504970/non-blocking-read-on-os-
pipe-on-windows?noredirect=1&lq=1)

The eventual solution using SetNamedPipeHandleState isn't really supported:
[https://msdn.microsoft.com/en-
us/library/windows/desktop/aa3...](https://msdn.microsoft.com/en-
us/library/windows/desktop/aa365787\(v=vs.85\).aspx) "Note that nonblocking
mode is supported for compatibility with Microsoft LAN Manager version 2.0 and
should not be used to achieve asynchronous input and output (I/O) with named
pipes."

------
JoachimS
fwiw wc -l in macOS Sierra 10.12.3 reports 1059 entries. There are quite a few
programs that are Apple specific (macbinary, appletviewer). But many of the
ones talked about in the article are here too.

------
floatboth
> date > /tmp/test_at.log will be executed at 9.26 PM on the same day

well that's not guaranteed on all systems :D On FreeBSD:

> at is implemented through the cron(8) daemon by calling atrun(8) every five
> minutes

------
jwilk
Sigh. /tmp is not your personal playground. Forget that it exists, unless
you're a security expert.

~~~
__Joker
Novice question. Why the temp folder is problematic? Some cursory google
search shows up this [http://security.stackexchange.com/questions/11606/what-
are-t...](http://security.stackexchange.com/questions/11606/what-are-the-
dangers-of-storing-webserver-temp-files-in-the-tmp-folder)

But generally what kind of attacks /tmp folder enables ?

~~~
JdeBP
Consider the headlined article, for starters. A simple

    
    
        ln -s /etc/passwd /tmp/test_at.log
    

which any unprivileged user can quietly do, before running its examples for
at, will ruin the system administrator's day just after 21:26.

This is the basis for many of the problems with files in /tmp : predictable
filenames written-to without care by privileged processes. It is a widespread
disease that is the reason that the systemd people arrange to run many
programs with PrivateTmp=true .

In this particular case, root's own home directory would have made a much
better private playground for the test_at.log file.

