
Linux Productivity Tools (2019) [pdf] - ketanmaheshwari
https://www.usenix.org/sites/default/files/conference/protected-files/lisa19_maheshwari.pdf
======
rietta
I do my development work and run most of my agency (with multiple employees)
with Linux. For some of the business things that most people don't assume
Linux can do, I use:

1\. Xournal to annotate PDFs (aka sign contracts without printing them and
scanning them back).

2\. LibreOffice of course for most document related work

3\. OBS Studio for recording webcam videos along with screen sharing

4\. Audacity for audio editing (heck, I used this even when I was on Mac OS X)

5\. Technically I've tried video editing with OpenShot, but do find myself
back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now

6\. pdfsandwich and Tesseract OCR for OCR/turning PDFs into searchable files

7\. Chrome/Firefox are both first class and run all the modern day web
application stuff

8\. Tons and tons and tons of command line stuff that Linux is well known for

9\. QEMU/KVM for hosting arbitrary virtual machines with almost native
performance

10\. GnuCash for double entry accounting for personal and volunteer society
finances. I used GnuCash for a while to run the S Corp accounting when we were
on Freshbooks and Harvest, but we've since graduated to QuickBooks Online for
better invoicing and CPA office professional services support.

11\. GIMP for photo editing

12\. Inkscape for messing around with vector graphics

Once you get past locating the tools to do your job. Linux has everything.

~~~
coronadisaster
> 5\. Technically I've tried video editing with OpenShot, but do find myself
> back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now

I wish that there was a better video editing alternative for a non-mac...
Openshot and the others I tried crash too much (Openshot crashed for me today
while trying to do simple trim)

~~~
Cactus2018
The new 2.5.1 version on OpenShot is a big stability improvement over 2.4
(which is the current repo version)

~~~
coronadisaster
2.5.1 is the version of openshot-qt that I was using

------
gen220
Its table of contents doesn't do it justice. this is a great deck! super dense
and straightforward.

my tiny tip to contribute:

for those who could never be bothered to remember the ctrl-commands for
traversing words, but are familiar with `vi`-style movements: you can go one
step above the suggestion in this deck with `set -o vi`.

~~~
_kst_
I'm a heavy vim user, but I still prefer the emacs-style key bindings for
bash. If I want to do vim-style editing on the command line, I sometimes use
Ctrl-X Ctrl-E to edit the line in my default editor (which is vim).

YMMV, of course.

~~~
BatmanAoD
`.inputrc` is your friend. I have `v` (in normal mode) configured to open the
line in an editor, because muscle memory sometimes makes me try to enter
visual mode when editing a command.

~~~
ketanmaheshwari
fc is an interesting related bash-builtin command. It will open $EDITOR and
put the last command in it to edit.

------
everybodyknows
_locate_ is not mentioned -- two orders of magnitude faster than _find_ for
searches from '/' on my laptop, e.g.

    
    
      find / -name \*bashrc\*
    
      locate bashrc
    

Faster because reading from a single binary cache file, updated nightly,
rather than opening every directory inode under /. To force an immediate
synchronous update of the cache:

    
    
      sudo updatedb

~~~
stinos
Relying on a database which gets updated nightly is problematic for some usage
though in at least 2 ways: I don't leave my computer on if I'm not using it so
there's a good chance the db will simply never update (I guess?) plus it's
only usable for locating files with confidence if you know they're system
files or else they were on the system yesterday. Or, you run updatedb manually
all the time and weep because that's not something instant, at all.

All in all, the idea is good, but I'm almost sure there are better
implementations (even Windows has tools like 'Everything' which use a database
but which gets continuously updated).

~~~
scbrg
There are cron implementations that handle the "turned off at night" problem.
anacron is one of them.

------
nextos
Nice slides. I missed him covering ripgrep and friends, which is very useful
to avoid slower find-grep queries.

Along with ripgrep, I think GNU Parallel (covered in the slides) and htop
(briefly discussed) are great additions to any Unix development environment.

The rest of the standard utilities have stood the test of time surprisingly
well. But top is a bit wonky, xargs has many pitfalls and, as I said, ripgrep
is great for speeding up some find-grep workflows.

~~~
gen220
+1 for ripgrep (it really _is_ a lot faster).

Also, I recommend using fd as a replacement for find. It’s API is a bit easier
to grok imo, and it’s similarly blazing fast.

fd/rg/fzf are the core tools I end up using most frequently, working in a big
monorepo.

~~~
fiddlerwoaroof
Does fd actually cover all the same ground as find? I used to be intimidated
by find, but these days I find it’s extremely useful for all sorts of
otherwise complicated file location operations. For example, to find files
newer than a certain date, just use ‘touch’ to set the date of a temp file and
then ‘find -newer temp’. Or, I have a script that deduplicates all the regular
files in a tree by hard linking them to a file named by the sha256 of the
file’s contents ‘find -type f | ( while read -r fn; do ...; done )’ and then,
after deleting files or whatever, I can “garbage collect” the shas by deleting
files in my link farm that only have one hard link ‘find -links 1 -print
-delete’

~~~
lozf
Much the same here, but I tried fd the other week, and for _simple_ searches
it definitely feels much quicker and "lighter", so worth adding to the toolkit
imo.

------
sirwitti
Here's my non-cli list of to me usefuly software on linux (working as
freelancer since 2010):

\- Webstorm + Phpstorm: Amazing tools for web development

\- Sublime text for simple text editing

\- Meld for graphical diffs between files and directories. Very useful.

\- Signal client: Using it more and more recently

\- Spotify: That's not productivity though :)

\- Joplin for markdown formatted note taking

\- KeepassCX + FF extension: Works decently for password management. Though
there's no solution for FF mobile as far as I know.

\- Reaper for professional audio production (never liked audacity and ardour).
There's also Bitwig which is more like Ableton live but quite expensive.

\- Gimp: I use it but I don't like the user experience.

\- Inkscape works pretty well for anything vector based

\- Libreoffice: I tend to use calc regularly

\- Darktable for raw image editing

\- Arduino

\- Figma for digital design (web-based, so not really linux)

\- Nextcloud as file storage and for synching contacts and calendars.

\- Zoom for video conferencing (need to try jitsi again though)

~~~
newsch
> KeepassCX + FF extension: Works decently for password management. Though
> there's no solution for FF mobile as far as I know.

Not sure what mobile platform you use, but on Android I have been very happy
with Keepass2Android
([https://github.com/PhilippC/keepass2android](https://github.com/PhilippC/keepass2android)).
It uses the android password-manager API so you get autofill (or at least a
quick-access button) on login fields in both apps and websites.

Edit: It also supports a lot of methods for syncing the keepass database
between devices, I point it to a directory that's managed by Syncthing and it
works pretty flawlessly.

~~~
sirwitti
Thank you, I will check that out!

------
d33
One useful thing when managing one's time is to know _when_ a command will
complete. I have a habit of using "pv" anywhere I wait for a result so that I
know how long it takes:

[http://www.ivarch.com/programs/pv.shtml](http://www.ivarch.com/programs/pv.shtml)

It's basically "cat", but with a progress bar. Also useful for measuring IO
speed and volume (e.g. decompression: `pv -c largefile | gzip | pv -c >
/dev/null` )

~~~
jniedrauer
You can also use it with dd, or even just redirect it directly to a block
device. I use it a lot for writing USB images. `pv image.iso > /dev/sdX`. Just
gotta be careful not to redirect it to the wrong one...

~~~
traviscj
Project idea: ML-based snafu-avoider that prompts for confirmation when past
invocations by other users in the community of that command resulted in
manifestations of regret[1].

[1]: screaming picked up by mic; detected maniacal CTRL-C'ing as immediate as
it is futile; permanent loss of peer relationship to the project's cloud
(indicating subsequent reformatting and/or booting-off-tall-building)

------
scep12
This is an awesome deck. I'll be bookmarking this as a great resource to pass
along.

Tangent: although the portability and consistency of PDFs is nice, I'd love a
browser extension that gave me a nicer view of slides that were exported as a
PDF. I find there's something more satisfying about discrete pagination vs
continuous scroll for these types of things.

~~~
Brakenshire
Are there any good parsers to convert pdf to html/epub? It would rely on the
pdf being well structured.

~~~
schaefer
Pandoc has worked for me.| it literally does everything.

------
dvirsky
The python utils section missed the very cool json.tool that can pretty print
JSON files (and exists on your OS if it includes Python):

cat /tmp/foo.json | python -m json.tool

[https://docs.python.org/3/library/json.html#module-
json.tool](https://docs.python.org/3/library/json.html#module-json.tool)

~~~
trishmapow2
Don't abuse cats, use `python3 -m json.tool < test.json` instead.

~~~
traviscj
Bit of pointless contrarian opinion from a long time cat|'r:

> Characters saved with "<"? 3. > Hours lost with accidental ">"? unbounded.

Preface: I'm confident on the shell & with my keyboard: I've been using shell
redirects for a couple decades, learned touch-typing up to around 80WPM, and
used to note-take math lectures live in latex. Yes, I have noclobber enabled.
But...

I can't promise myself I'm on a machine configured that way OR that I won't
typo this one on a file that really matters.

I kinda use of `cat` as a `immutability-please` command: it _creates_ a
barrier/seam between this file and whatever I'm doing with it, such that I'm
_guaranteed_ nothing downstream will mutate it (as long as everything only
works on stdio).

In a kinda similar way: by starting with `cat x |` (or even just `cat x`,
frequently!), the rest of that compound command is _input source independent_
(and even _ignorant_!) -- if I want to hook it up later to a `curl
example.com/xyz.json`, I don't need to split it into separate
commands/introduce temp files/etc. This is especially because I find myself
doing a _really_ quick scan of the entire file ("did it return the actual
object this time or just `{}`?"), then piling on transforms from there.

Something about doing `cat x | head` and then kill-wording to fill in `next-
thing` feels faster and more consistent than `head x` and then `next-thing x`,
maybe because the filenames are often much longer than the commands? Maybe I
need a "go to start of line, forward-kill one word" hotkey?

tl;dr: i'm playing psychological games with myself and TBH not sure I'm coming
out ahead.

~~~
hikarudo
Wow, you described my workflow exactly. Using "<" might be shorter and faster-
for-the-machine, but it's less readable and breaks the left-to-right order. I
also frequently start with `cat x`, see what's in the file, move on to `cat x
| some filter` etc.

~~~
smorrow
'<file' can be used anywhere in the command. If it makes more human sense to
have the file near the start, you can do that.

grep <file -opts 'regex'

~~~
traviscj
This is why I walked back my claims a bit in my tldr — basically started
trying stuff out and had a “whoa not used to the way that looks, but it does
_work_...” moment.

I think I still have the “wrong” preference, but I appreciate learning some
new stuff!

------
ufo
Some distros don't install lsb_release by default. It might be better to
consult the /etc/os-release file instead.

[https://www.freedesktop.org/software/systemd/man/os-
release....](https://www.freedesktop.org/software/systemd/man/os-release.html)

[http://0pointer.de/blog/projects/os-
release.html](http://0pointer.de/blog/projects/os-release.html)

------
mszcz
These lists always delight and frustrate me at the same time. Every piece of
software seems to have a lot of those useful, little tricks, shortcuts and
functionalities but discovering them is always so difficult.

Biggest problem is that you don't even know they're there. You don't even know
to look, what to look for. I seem to always stumble my way ass backwards into
them, and then, be somewhat irritated that I didn't know them before.

------
Gasp0de
I like to use systemd timers rather than cronjobs nowadays.

~~~
Cactus2018
For those looking for a link
[https://wiki.archlinux.org/index.php/Systemd/Timers](https://wiki.archlinux.org/index.php/Systemd/Timers)

------
vietjtnguyen
feedgnuplot [1] is wonderful for quick visualizations of data from the shell.
Also great for decoupling visualization from CLI programs.

[1]:
[https://github.com/dkogan/feedgnuplot](https://github.com/dkogan/feedgnuplot)

~~~
kristopolous
Hey that's my old roommate! Check out his vnlog project as well, it's kinda
feedgnuplot 2.0

------
vz8
How in the world did I go this long without using the "at" command? (slide 69)

Thanks for the deck, OP.

------
earthscienceman
Wow. Well, I wish I would have known abut GNU parallel before I wrote that
massive bash script a few years ago. Thanks for posting this.

~~~
betaby
xargs can do most of the common tasks perl parallel does, see xars -P

~~~
RMPR
Imho, the main advantage of Parallel is that it can execute the tasks on
multiple hosts, straight from the man page:

> GNU parallel is a shell tool for executing jobs in parallel using one or
> more computers. The typical input is a list of files, a list of hosts, a
> list of users, a list of URLs, or a list of tables.

------
seanwilson
In native Mac text editors, you can use e.g. SHIFT + ALT + LEFT to select the
whole word on the left of the cursor. Is there a way to get a Mac terminal
that uses similar shortcuts? For copy/paste also?

Related, but I found [https://micro-editor.github.io/](https://micro-
editor.github.io/) which tries to use shortcuts you'd expect in native Mac
apps.

~~~
fiddlerwoaroof
There’s a couple options here: most of the Mac terminals allow you to specify
custom key-combo -> Terminal input mapping that can be used to get this. Also,
on most shells you can customize the interpretation of the key bindings to
attach this key combo to a specific editing command. Which works best depends
on the functionality and the key combo you’re trying to bind.

------
chewxy
slides 14 and 15 - terminals have emacs shortcuts??!

EDIT: I am god now

~~~
anarchyrucks
You can get vim shortcuts as well.

zsh: bindkey -v

bash: set -o vi

~~~
filterfish
You can configure readline (which is what bash uses) to use vi-bindings via
$HOME/.inputrc then everything that uses readline (which is pretty much
everything) will also get vi-bindings.

------
dav
The main thing I miss from OS X is Droplr (pain free sharing of annotated
screenshots) and Jumpcut (pastebuffer history tool). Can't find anything on
Linux for either of those that works nearly as well. I can still go to an old
Mac for occasional needs like Final Cut Pro but those are tools I used
several/dozen of times every day before.

I'm on PopOS.

~~~
brightball
Have you tried Flameshot for screenshots?

~~~
dav
I had not, just installed it though and it looks fantastic, thanks! The best
I'd found previously was a no-annotation script that failed most of the time
with a vague "something went wrong" error.

~~~
brightball
I hunted around for a while looking for something similar. I spent so much
time in Slack working remote, that fast screenshots with arrows and notes are
critical to communication.

Coming from a Mac environment, I can't remember what the tool was that I used
previously but I think it was purchased by Evernote. Flameshot has been
excellent as a replacement.

~~~
dav
You're thinking of Skitch, I used that too before Droplr.

------
wiz21c
Not exactly linux but the Speedcrunch calculator is really good.

------
matthewhartmans
Another good productivity hack is having a good clipboard manager.

I highly recommend Green Clip:
[https://github.com/erebe/greenclip](https://github.com/erebe/greenclip)

You can assign custom shortcut keys to pull it up and it can fuzzy search
through your clipboard items.

~~~
RMPR
Clipboard managers are probably the thing I miss the most on Wayland.

~~~
c1c2c3
Haven't tried this yet but worth looking at?
[https://github.com/yory8/clipman](https://github.com/yory8/clipman)

------
blt
This is great! I might add a slide or two about Makefiles. Probably the topic
is too big and out of scope, but inexperienced Unix users may not realize that
they are useful for so much more than compiling multi-file C programs.

~~~
kabacha
Please lets not recommend anyone new to Makefiles. It's such a dated system
that you need to use tabs for functionality.

Just one of million better alternatives like
[https://www.pyinvoke.org/](https://www.pyinvoke.org/)

Makefiles need to be allowed to die so we can clear ourself of this absurd
legacy cruft.

~~~
dima55
Make is for tracking dependencies between targets, and rebuilding stuff. Looks
like pyinvoke does some glorified scripting, or something. I will never
understand why so many people have no idea how to use Make, but are convinced
that it is terrible, and then try to fix it by reimplementing some tiny subset
of its functionality.

Make is great. Learn it an use it. Its limitations are well-known, but easy-
enough to work around. Fixing them properly requires lots and lots of
complexity, and you get bazel when you try. And it's way overkill for all but
the biggest projects, which is why Make has been so successful.

~~~
kabacha
It's was great but then we improved our work flow patterns. Make is really
limited and ugly by today's standards. I mean you have to use tabs for Pete's
sake and if you include a single space the program will break. It's dated
software that should be abandoned with either evolution or revolution.

------
hofstee
Other useful tools include mosh or EternalTerminal to handle either low
bandwidth or spotty network when you need to remote into a server, as well as
VNC for when you need a GUI on the server but X-forwarding is too slow.

------
necovek
I expected something more of a traditional set of "productivity tools".

Eg. a tool like "PDF Mod" is a gem I struggle to remember the name of when I
set up a new computer — quick and easy way to assemble/extract PDF pages :)
It's long unmaintained it seems, though.

For those expecting the same, this is mostly "GNU terminal tools" (+ python,
tmux and a few other tidbits), so applies to pretty much any modern GNU system
(including non-Linux systems, and I am pretty sure homebrew on Mac has all of
these).

------
kayali
Thanks, these look great. Is there a recorded talk for these slides?

~~~
ketanmaheshwari
Unfortunately no. The conference recorded only regular track talks. This was
on Tutorial/Training track.

------
aesyondu
> cmd #tag to tag a hard to remember command

Command tagging wasn't really explained in the deck. I had to read this
archived web page for an explanation:

[https://web.archive.org/web/20191216131445/https://www.ostec...](https://web.archive.org/web/20191216131445/https://www.ostechnix.com/apply-
tags-to-linux-commands-to-easily-retrieve-them-from-history/)

~~~
ketanmaheshwari
Thanks for the comment. An example of #tag in action is shown in slide 39 (In
the talk, I mentioned we will see an example later).

------
cookiecaper
This is an awesome intro, thanks! Will definitely share.

I can't recommend fish-shell highly enough for anyone spending significant
time in the CLI. It can't completely replace bash et al, but it's not really
meant to. Just install it and give it a try and you'll get addicted pretty
quickly.

------
enriquto
The only thing that I miss in linux is a pdf viewer that can show inline
animations (like those produced by LaTeX's "animate" package). I am still
forced to run adobe's 32 bit pdf viewer because it is the only viewer
available in linux that supports this kind of pdf.

~~~
BatmanAoD
Wow, that seems like a rather questionable feature for a "portable document
format".

~~~
enriquto
If you use pdf for making slides for a presentation, it is a legitimate use
case to want some automatic animations or videos. It is actually possible, and
adobe's viewer certainly supports it, but no program on linux does.

You could argue that we shouln't be using pdf for making slides in the first
place, and maybe you are true. But if I want beautiful math typography (which
is even more important than animations), all other options are sadly not up to
par.

~~~
BatmanAoD
I understand; I used Beamer back in college. It just seems like that's not
really a good fit for the original purpose of PDF files. If they wanted to
support presentations, maybe they should have created a new file type based on
PDF but with extra capabilities.

------
NotUsingLinux
I guess this is not the place to discuss about how to change all of computing
and how it should be done better and that Linux is a distraction?

[https://www.youtube.com/watch?v=rmsIZUuBoQs](https://www.youtube.com/watch?v=rmsIZUuBoQs)

------
arendtio
If you want to know which cli tools you can expect on every *nix machine, I
can recommend this page [1] which gives a nice overview of the POSIX shell
standard.

[1] [https://shellhaters.org](https://shellhaters.org)

------
erikbye
Some essential stuff lacking here (unless I missed it): pushd, popd, dirs,
jobs, and fg.

~~~
ketanmaheshwari
I did have jobs, fg, nohup in an expanded version but had to trim to fit the
slot. I considered pushd, popd but decided against it as it requires a bit of
explaining -- figured I won't have much time.

~~~
erikbye
I see. I tend to recommend zsh, anyway, which has navigational features that
superannuates pushd, popd, and dirs.

------
dan_can_code
Awesome! Thank you for sharing this. I've been trying to get better with my
Linux CLI knowledge, and this will really help.

------
thisisnot
As a wsl user I miss all this tools.

------
vnxli
Is the presentation this was delivered to recorded? I'd watch whatever lecture
this was from.

------
tomerbd
Is there any link to the actual video presentation of this?!

------
antb123
my list visual stuff:

\- xournal - pdf editor

\- calibre - epub

\- keybase - slack replacement

\- cherrytree - notes

\- anki - study

\- shutter - screenshots

\- zoiper - voip calls

\- kazam - video snippets (screen recording)

\- remmina

\- vsc

command line

grep -nr, locate, ncdu, htop, ipython, vim, ssh, tsp, tail, less, youtube-dl
;-)

------
hacknoid
Great compilation

------
ChankeyPathak
tmux ftw!

~~~
nanna
screen!

------
gerdesj
I absolutely love this sort of thing. I am a sysadmin with something like 30
years experience. I am as passionate about the subject as anyone but I lack
some presentation skills and possibly imagination. I'm also old enough to know
what I lack!

Why not give open source stuff a go?

------
jcpst
My poor brain corrected the url to “unisex.org” on first glance.

------
seemslegit
Otoh facilities like Oak Ridge can be expected to be conservative in their
tech, otoh sed and awk should be declared as a threat to national security.

------
sytelus
How much of these is really still relevant with now Python quickly becoming
new scripting language across all platforms? I now tend to prefer 3 lines of
Python vs 1 line of cryptic unmaintanable Linux-only bash. With new great
subprocess module in Python, I feel bash scripting is really unnecessary
except in few situations like .bashrc or really quick onliners.

~~~
jpxw
I don’t think Python and bash solve the same problems at all. I don’t see how
they could be seen as comparable, much less interchangeable.

~~~
II2II
If you're doing any filesystem based manipulation, then it's probably easier
to use the shell and tradition Unix utilities. If you're manipulating files,
the sort of thing where awk and sed start becoming important (particularly
complex manipulations with awk and sed), you're probably better off with
Python. Those use cases are pretty much why languages like Perl and Python
came to be.

------
kabacha
I feel that most of the tools mentioned here are quite outdated already and in
2019 alternatives should be recommended.

find -> fd

grep -> ripgrep

wget/curl -> httpie

awk/sed are just horrible to learn for beginners, would advise to just pick up
python instead via xonsh[1] or pyp[2]

How about we don't teach new users these legacy tools and we can finally
progress a little?

1 - [http://xon.sh](http://xon.sh)

2 - [https://github.com/hauntsaninja/pyp](https://github.com/hauntsaninja/pyp)

~~~
oehtXRwMkIs
As much as I like httpie, in my view it's not a strict improvement like fd or
ripgrep. This is unfortunately due to it being written in python.

~~~
gitgud
Well I use it all the time and it's a huge improvement to my life. The elegant
and simple syntax makes testing endpoints so much easier than the verbose curl
and wget alternatives...

curl

    
    
       curl --header "Content-Type: application/json" -d '{"some": "content"}' <host>/<resource>
    

httpie

    
    
       http <host>/<resource> some=content

