I found jq the other day (http://stedolan.github.io/jq/), which is a commandline utility for manipulating json data, resembling awk.
Considering the number of API's that return json data these days, it's simplified my life a great deal. It's proven especially useful for automation shell scripts, when working with things like the aws command line client that return all results as json. Usually I'm looking to extract something like a single field to use as input for a following command, and jq makes it really easy. No more inline python or awkward sed regex's. :)
It's available using apt-get on debian/ubuntu, but that version is out of date with the online docs. It's rather trivial to build from source though (it has no external deps).
On a similar note, there is also csvkit [1] for csv files. It is not quite as full featured as I had like, but it is easier than manipulating csv content directly via cut/column/grep/sed.
Exactly - OS X is Unix(tm), but davexunit's complaint is that the term "Unix commands" implies "commands that will work on all/most UNIX/POSIX systems", not "commands for a single OS that happens also to be a UNIX".
I agree with the complaint too but not only are many of these commands mac only but in addition to that, they are not less commonly used (ex vim). I feel like I could have gotten quite a few less commonly used tools by
ls /usr/bin/|sort -R|head
In my opinion it was a poorly named article with just as poor content. Many of the descriptions of the commands were not good ("lsof: godlike"). Better descriptions could generally even be found automatically:
Which most likely only happens because the majority of HN users only know either GNU/Linux or Mac OS X as UNIX and never really used the big commercial ones.
Every time someone mentions here how POSIX and C are portable across UNIXes, I have recolletions of maintaining code full of #ifdefs to make it work across such "portable" systems.
A very useful curl switch that I discovered, resume of partially downloaded file ! -C or --continue-at <offset>. When offset is "-", curl will use the output file to figure out the offset.
curl -C - -o "some_file"
It will basically get the size of "some_file" and use it as an offset when making the HTTP request, similar to wget --continue.
Very useful to download a file from a unreliable network and when you need all those security cookies. To get the cookies I use "Copy cURL" from Network pane of Chrome's "Developer tools".
When I'm on a bad connection, I like to download the file on a server somewhere, then rsync it locally. As long as my server and the server I'm downloading from are on fast connections, the initial step doesn't take long, and this frees me from worrying about expiring cookies or URLs or whatever.
Most people know only the bare minimum to move files around. Not everyone has spent their life on a nix command line.
And I don't even know why you'd think iostat, watch, htop, or tmux would be well-known amongst anyone other than sysadmins and long-time nix power users. And tmux didn't even exist five years ago.
Fair enough. 5 years is a long time in the tech world to me. Tons of things change in that time. I'm not a sysadmin, and I'm not against educating people about these tools. I'm also not saying the exact inverse of the original statement (that everybody knows these). What I am saying is that it's a stretch to say they aren't well known. Call it "intermediate linux usage" or something.
vim and xargs are less commonly used commands? Seems to be a mix here of small unix-y utilities and larger programs (like vim and tmux). Good stuff to know though.
Nope.. Haven't rebound anything. But strangely I very rarely need either of them, so haven't bothered so far to remember the shortcut (however I would indeed rebind them if that were to change).
However.. point taken. The default bindings for those two functions are terrible (at least with a german keyboard layout).
Well, you never have to, but it can be useful, e.g. with C-M-x instead of M-x eval-defun RET. I've never yet had to bind a chord with more than two meta keys, but considering that even Photoshop has three-meta chords (e.g. C-M-S-s, File/Save for Web...), I don't really see the problem.
I got used to "open" on the command line on my late, lamented iBook. Now whenever I'm in Linux I put "alias open=xdg-open" in my .bashrc and it's pretty much the same thing.
Eh. wget is described as "get w's", which is funny, and it's not as though 'man wget' is a long way away -- even Google knows what to do with that input. The same, I'm sure, is true for 'man lsof'. That being true, I don't see any reason for the author not to indulge in a bit of humor here.
One that I have recently been using more and more is time, but don't forget there is a difference between bash's "time" command and /usr/bin/time . I use both.
The majority in the list seem pretty common, but I guess with more and more gui dependents some people just never spent days wading through man pages or info coreutils, so still useful.
watch: execute something on a schedule *in realtime*
printf: change the format of output
apropos: info on commands
dd: disk writing
ss: socket statistics (show apps *using the Internet*)
Right, that's what I'm saying. That's enough information for me to decide whether I need to google further if it's something I need to install myself (or use man if it's installed out of the box). The examples I gave don't give me that first bit of information.
comm is the one I always forget the name when I need it. Because it's not cmp/diff but the idea is similar
Very nice for finding differences between files (like config files, or results of some tool) where diff won't help you (too big a diff, or files are too different)
Site's non-responsive now, wonder what the list was?
I'll vote for "yes" which at least seems intuitively useful in a "yes | rm -i something" kind of way. Well that specific example is useless... Also over two decades ago if you obtained access to someones account it was considered highly amusing to end their .profile with a call to /bin/yes. Oh and before the newfangled /dev/urandom (or /dev/zero) we used to redirect "yes" out to media to wipe them or test overall system thruput.
Now for one thats truly obscure and ripe for abuse because no one ever runs it anymore, try RMT the remote magtape manipulator.
column: create columns from text input
tr: translate/substitute/delete input
join: like a database join but for text
comm: file comparison like a db join
paste: put lines in a file next to each other
rs: reshape arrays
jot: generate data
expand: replace spaces and/or tabs
time: track how long a task takes to run
watch: execute something on a schedule in realtime
xargs: execute something on all inputs
iftop: visually show network traffic
htop: show system stats more powerfully
xxd: manipulate files in hex
mtr: powerful traceroute replacement
mdfind: osx find replacement that uses spotlight
brew: osx package manager
df: disk usage
du: disk usage
dig: dns queries
host: dns queries
man ascii: lookup your ascii
sshfs: mount a directory through ssh
wget: get w’s
tmux: a better screen
pushd: push your pwd to a stack
popd: pop pwd off your stack
lsof: godlike
ncat: use this instead of nc
fuser: kills locking processes
vim: attack yourself
tac: cat in reverse
rename: change spaces to underscores in names
open .: in osx, open finder in the current dir
lsmod: show kernel modules
printf: change the format of output
timeout: execute something and kill it soon after
disown: protect a job from disconnect
fc: edit your last command in your editor and execute it
sudo !!: repeat last command with sudo
tee: send output to stdout as well
pgrep: greps through processes
pkill: kills processes based on search
fmt: text formatter
bc: an interactive calculator language
apropos: info on commands
strace: the uber debug tool
man units: interesting
pstree: shows processes in a…well…tree
ddate: wtf
zgrep: grep within compressed files
zless: look at compressed files
readlink: read values of links
atop: another top
split: break a file into pieces
dd: disk writing
ndiff: show differences in mmap scans
ss: show apps using the internet
whatis <cmd> when unsure of what a command does is quite helpful and displays short manual page descriptions, for instance some metaphysical question:
> whatis time
It would be very useful to delineate this list by command type and availability. The ones I wanted to check out (ndiff, rs, iftop, mtr, ...) must be in packages, since they don't respond in either OS X or vanilla Debian.
Maybe your friend is thinking too much about enterprise set ups. I have no need for sshfs at work, but for my personal servers that's a different matter. I find it invaluable to have the convenience of being able to have an NFS-like share over a simple and secure SSH socket. Yes you could literally run NFS over VPN or an SSH tunnel, but sshfs is just so much more convenient.
So really it just depends on what you're trying to do as to the practicality of sshfs. But then I guess you could argue that is the case for nearly all the commands that many sysadmins have flamewars over.
i also tried to use pop/pushd ...tried hard. and never got it integrated into the work flow. however this guy's method has saved me a lot of keystrokes (very handy if one has a few, frequently visited, yet deeply nested, directories): http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-f...
Using some aliases you can have pushd automatically. What then ends up useful with that is you can use ~1 ~2 ~3 ... to refer to old directories. That will change your life :)
Considering the number of API's that return json data these days, it's simplified my life a great deal. It's proven especially useful for automation shell scripts, when working with things like the aws command line client that return all results as json. Usually I'm looking to extract something like a single field to use as input for a following command, and jq makes it really easy. No more inline python or awkward sed regex's. :)
It's available using apt-get on debian/ubuntu, but that version is out of date with the online docs. It's rather trivial to build from source though (it has no external deps).