Hacker News new | past | comments | ask | show | jobs | submit login
MIT Hacker Tools: a lecture series on programmer tools (hacker-tools.github.io)
1028 points by anishathalye on Feb 4, 2019 | hide | past | favorite | 79 comments

Hi all! We (@anishathalye, @jjgo, and @jonhoo) have long felt that while university CS classes are great at teaching specific topics, they often leave it to students to figure out a lot of the common knowledge about how to actually use your computer. And in particular, how to use it efficiently.

There’s just no class in the undergrad curriculum that teaches you how to become familiar with the system you’re working with! Students are expected to know about, or figure out, the shell, editors, remote access and file management, version control, debugging and profiling utilities, and all sorts of other useful tools on their own. Often times, they won’t even know that many of these tools exist, and instead do things in roundabout ways or simply be left frustrated about their development environment.

To help mitigate this, we decided to run this short lecture series at MIT during the January Independent Activities Period that we called “Hacker Tools” (in reference to “hacker culture”, not hacking computers). Our hope was that through this class, and the resulting lecture materials and videos, we might be able to bootstrap students’ knowledge about the tools that are available to them, which they can then put to use throughout their time at university, and beyond.

We’ve shared both the lecture notes and the recordings of the lectures in the hopes that people outside of MIT may also find these resources useful in making better use of their tools. If that turns out to be true, we’re also thinking of re-doing the videos in screen-cast style with live chat and a proper microphone when we get the time. If that sounds interesting to you, and if you have ideas about other things you’d like to see us cover, please leave a comment below; we’d love to hear from you!

We’re sure there are also plenty of cool tools that we didn’t get to cover in this series that you all know and love. Please share them below along with a short description so we can all learn something new!

Anish, Jose, and Jon

That's funny, I recently had the same revelation (that it's hard to discover these things) with engineering students in mind. I wrote an e-book called Digital Superpowers, covering many of the same topics you did. I added some general interest things on art and publishing too. I think I like your approach better.

Anyway, my list of topics is as follows, maybe you want to consider adding a few of them to your list?

Password Managers • 2-factor authentication • Virtual machines • Package managers • Command line • Routers • QR codes • Port forwarding • Python • Pandas • Onion routing • Ad-blocking • PDF reordering • Flow charts • Regular Expressions • Time-lapses • Encryption • Column edit • Digital photo editing in RAW • Audio noise reduction • Vector graphics • Making websites • DJing • LaTeX publishing • Reference management • pandoc • git version control • Programming • Web apps • Self-hosting • Home automation • Monte Carlo • VPNs, • ImageMagick • Linear regression • ffmpeg • Sphinx • Robotics


That's the one?

EDIT: I guess it is, via https://digitalsuperpowers.com/. The list of topics is great, gonna check the book out!

That's it! Let me know how you like it. It's my first foray into writing books. Walking the line between utility and breadth is tough.

As a computer Science graduate with no experience doing images that section alone was worth getting the book for me. I plan on using at as encyclopaedia for future uses.

I am happy you had compatibility in mind as I am not super fussed on which operating system I use.

I would suggest adding a virtual desktop section as that has been the biggest gamechanger for me ( I use windows 10s with virtual desktop enhancer) I dont know any linux or mac equilivants though.

I'm pretty confident in claiming macOS and many Linux desktop environments natively support virtual desktops. It's really only Windows that doesn't.

Glad that was useful, thanks for saying so! Great idea on virtual desktops.

Bought it! I'll shoot you an e-mail with my thoughts once I'm through.

Hopefully you change the recommended tools based on the OS. (i.e. OSX would be slightly different from linux for certain commands). And focus on doing as much as possible from the command line?

I tried to highlight examples that are available on Mac, Linux, and Windows through the package managers, (homebrew, apt, chocolatey) and at least provided system-specific installation lines for each based on those. It is highly command-line and pure-text centric.

this is insanely ambitious, and in a weird order

You're right. And "covering" is a strong term for the book. It only introduces them, offers a few exercises, and moves on. It's still 200 pages. It's intended to help engineering students (mechanical, civil/structural, nuclear, etc.) discover these wonders and then choose which ones to go deep with based on their needs and interests.

Thanks for sharing and seems it would be useful for non-engineering students as myself as well. Going to check it out =)

> ...[universities] often leave it to students to figure out a lot of the common knowledge about how to actually use your computer.

Now I'm out in the real world (MIT class of '86!) I think this is more of an MIT thing -- many graduates of other schools get explicit training in things that the MIT faculty, fairly or not, assumes students will simply pick up. And this is despite the heavy emphasis on actually doing stuff, especially in the school of engineering. Actually I've had hit or miss results hiring freshly-graduated MIT grads (maybe less so VI-A) for this reason, though once they are up to speed they've mostly been great.

Really good use of IAP by the way!

> many graduates of other schools get explicit training in things that the MIT faculty, fairly or not, assumes students will simply pick up. And this is despite the heavy emphasis on actually doing stuff,

uh...I graduated from a low tier state school that was notable in our state for focusing on practical matters....and we didn't get 'explicitly trained' on this either. Most people do just pick it up.

I got a CS minor from a top-5 program, and while a lot of the core classes definitely covered some of these topics in various degrees, it's really nice to see all of these aggregated in a concise way.

While you definitely do pick stuff up over time, this is arranged in a way that I could see myself preferring to reference this rather than find that one stack overflow answer that clicked with me at the time. Now that I at least have a better understanding of these concepts 3.5 years into real-world web development, this seems like a good resource to cover my bases better now.

Well, I appears from my experience a surprising number of MIT students don’t, making this mini-course popular.

As an MIT grad myself I should be biased to think otherwise, and I certainly hope the credential stands for something good. But I haven’t found much difference in programmers 5 years out of school — it’s been attitude that has mattered more.

>many graduates of other schools get explicit training

From my undergrad programming class, which was a while ago, I remember that the first class had a brief intro to vim. But something like vim is an acquired taste. I never used or appreciated vim until much later when I saw others at work use it. During school, I think I used geany since that's relatively easy to use and install on ubuntu. So maybe there is some merit to spending more time on tools. However, my experience is from before everything was so easily accessible on the internet (i.e., pre github, stackoverflow etc.). So maybe these days it's not such a big deal.

I graduated from a low-tier state school and after reviewing MIT's course materials along the way, I feel the opposite. Sure there wasn't a class, but plenty was covered explicitly in the homework and labs.

All I can say is, despite my pro-MIT bias, my experience has been at best mixed, on this dimension.

(BTW, among things I’m glad that MIT has done, open courseware is top of my list. Glad you used it!)

University of Washington CSE added a practical course to get students familiar with Linux and C partly because students are expected mostly to come through other departments / disciplines where the only programming may have come from Java or Python. Students were having great difficulty in higher courses like graphics, OS, compilers, and embedded systems because we weren’t teaching programming courses in our curriculum except maybe our programming languages course itself which covers languages like ML, Scheme, Haskell, or Eiffel and OOP languages like Smalltalk.

Early pilots of the course were put together with a set of Linux guides including bash scripts and Makefiles, source control with svn (hopefully hit by now), and moved quickly into practical stuff with bison / yacc to parse complex files and work with GCC output. I never got to see how well it was received by students but I was glad to see students go through the time-honored traditions of learning CLI tools.

Will need to take a deeper look later but from a quick skim I really like the approach taken: a well curated list of time tested tools across each area, going deeper where applicable.

For me the key point above is about a lack of tool knowledge causing "roundabout approaches" and frustration. Knowledge of these tools can do so much to demystify and make the seemingly arcane start to slot into place.

Do you like that kind of talks ? https://www.youtube.com/watch?v=sCZJblyT_XM

Gary Bernhardt - The Unix Chainsaw

Thank you so much for sharing your knowledge with the world. I'm excited to follow along with this and I love your Rust streams.


Thanks a lot for this! It appears like the recording doesn't really capture the screen well for multiple videos. I wish it was more professionally done, it could've been useful for years to come.

For most of the videos, we do have screen recordings (e.g. see https://www.youtube.com/watch?v=VW2jn9Okjhw).

Sadly, we're missing screen recordings for a couple videos (such as the shell video, where we had some issues with the screen recording software). We're thinking of re-doing some of those videos in a screen-cast style so that they'll be more useful for viewers.

The equivalent UCLA course is CS35L: Software Construction Laboratory (https://web.cs.ucla.edu/classes/winter19/cs35L/). It's taught by Paul Eggert (big open source/coreutils/emacs contributor + author of diff/sort).

Paul Eggert is also the maintainer of the timezone database and designer of the uniform naming convention for time zones, such as America/New_York and Europe/Paris.

What a champ!

Damn I wish that class was open access.

Here’s the CMU equivalent course, which interestingly was also designed by students:


CMU did (maybe still does?) have a mandatory, once-a-week course to orient new students to logging in to *nix workstations, setting file permissions and copying files around on AFS, etc. Whether this was just for CIT or university-wide I cannot recall.

Didn't they recently nuke the unix infrastructure and replace it with cloud services? Or have reports been exaggerated?

The unix infrastructure is still in place and widely used by nearly all cs courses.

Please note that the Software Carpentry [0] project exists as well. All welcome .

[0] https://software-carpentry.org/

MIT went with the Berkeley editor. A lost opportunity to introduce programmers to the beauty of using Emacs and Lisp.

But almost all IDEs and editors have a VIM mode - learning VIM is a good way to teach people how to keep their fingers on home row, and the skill is (because of what I stated previously) easily transferred to whatever editor you like.

From my perspective, I tried to learn Emacs, and decided it's a great editor, once you spend weeks to months to get it actually "working good," and learn shitloads of commands (let alone keybindings). Given my initial goal was "keep hands on home row," it was far, far faster/better for me to spend 3 days learning/memorizing VIM commands, installing a vim plugin into VScode, and just doing that.

Would love to use Emacs, but it's a nightmare to set up. I still haven't managed to get it feature parity with vscode.

> Would love to use Emacs, but it's a nightmare to set up.

Maybe you would like spacemacs then? It is a Configuration Bundle for Emacs focusing on "Evil Mode" (aka the Vim emulator in Emacs). It is just a few minutes to install and if you already familiar with vim then there should not be much change what the normal editor usage is concerned. Of course the customization is Emacs lisp which has a bit of a learning curve, but just to get started with Emacs and evil you don't really need to customize anything.

I wouldn't say Emacs is "a nightmare to set up"; rather, it's a continuous process of learning and improving.

Getting Emacs installed and running should hopefully be easy (e.g. via your OS package manager). After that I think it mostly comes down to coming up with workflows that you're comfortable with, building muscle memory and gradually introducing new things to see whether or not they stick.

I’m sure a not-insignificant portion will move on to Evil mode.

Vim -> Emacs is a better direction than Emacs -> Vim, because you’ll learn the more efficient editing style, then have access to the better environment, instead of getting used to the great Emacs environment minus the Vim keybindings, and feeling constrained by losing this when moving on to Vim.

I think this is true. I learned the Emacs keybindings and now they're muscle memory. I can't transfer that to Vim, but I could have transferred the other way and be proficient in two great editors.

Nobody wants to die on Emacs Keybindings Hill, even diehards like me and my pinky.

When I took C at University of New Mexico, and later Data Structures and Algoithms from the same professor, Emacs was required. Period. He even provided a download of his .emacs file (still tucked into my init.el). He also stressed make, gdb and valgrind ("It's not fully clean until it's valgrind clean.")

No emphasis on git, but I used it anyway.

A decade later, and my Emacs easily surpasses most VS Code features.

A decade later. Heh. Co-workers use VS Code OOTB and I'm like, "Yeah, but can you tangle or play Tetris?!"

Truth is, whatever works for you and gets out of your way is the right tool. I'm invested and happy with Emacs, but also I get why it's not the first choice for a lot of devs.

Even though it should be. :)

As an MIT alum I was sort of surprised they went with vim over Emacs too. MIT is almost the Vatican of lisp, after all. I sort of felt like an apostate for switching over to vim there.

Can't tell whether this is sarcasm or not. Perhaps the expectation is that those things will be acquired naturally; certainly I was immersed in both within two weeks of arriving at MIT. (Admittedly that was in the latter half of the Carter administration; things may be different these days.)

This looks good. I would also recommend the Software Carpentry program:


This is a great series. I'm going to share with my team, just to make sure they can fill in any items they're missing, like using bash functions for shortcuts or aliasing.

BTW, Stanford did an online training course 5 years ago or something called CS 184 Startup Engineering. They had some excellent tutorials on the CLI and Emacs [1]. -I wish I could find the materials, if anyone else can find them.-

Actually, I found them online: [1] http://docshare.tips/cli-git-emacs-dotfiles_57592eb2b6d87f37...

[2] Link to their dotfiles for beginner config: https://github.com/startup-class

So one lecture covers window managers for tiling. I used dwm for years and love it, but am stuck on Windows at work. Is there anything for Windows 10 that has a similar approach of having a stack with your unused applications and a big window for the app that is in focus? And then you can easily move window from stack to focus and vice versa?

I looked at the Wikipedia article [1], but none of the alternatives seem to do that.

What can you recommend?


[1] https://en.m.wikipedia.org/wiki/Tiling_window_manager

~Year ago, I had plenty of success with bug.n - https://github.com/fuhsjr00/bug.n. It's a DWM clone written in... AutoHotkey, of all things.

Ok,this blew my mind in thousands pieces. I'm going to try it right away.

You could have a full-screen Xming session connected to dwm running in WSL, if you liked that better. Of course, you would still have to use the standard Windows DE/WM when using Win32 applications, but if most of your work is text editing, web browsing, etc, you might be well served by the full-screen Xming.

You could start with this guide, which solves a slightly different but related problem: https://virtualizationreview.com/articles/2017/02/08/graphic...

It's no where near complete like dwm for that, but it is highly customizable, I use AquaSnap when I have to use windows for work. It is paid but I haven't had any issues with upgrades or redownloading the software when I haven't had to use a windows machine in a while.


If your stuck on windows, give MobaXterm a try. It's the commandline Windows should have had.

Sorry, but I don't think there is anything you can do about the GUI..

I like to use x2go on another machine or VM, it gives you access to individual windows/tiles for Linux programs, similar to Citrix published apps.

Haven't tried it but I think HashTWM[1] claims to have a similar main/stack model for windows.

[1] https://github.com/ZaneA/HashTWM

I strongly believe man is a product of his tools, my slow but steady increase of their mastery is a source of pride. I've spent nights just mindelessly browsing the web in the search for a tool I didn't know about (it can be a service, a browser extension, a text editor plugin) so just looking at the title of this post made me ready for an injection of dopamine.

I have a branch on my todo list solely focused on my tools, because there are so many things to configure continuously. I schedule these tasks for the occassion I feel like I'll have a couple of minutes to improve and their priority are often labeled as "Compounding Rewards" as opposed to "Critical".

Love it! Do you have any favorite tools you'd recommend, and/or favorite places for finding new ones?

I'm interested if you have a list of your favorite tooling resources?

We had a student run course at Purdue for a while to teach new students fundamental tools outside of class. https://github.com/Purdue-CSUSB

Really useful! I was looking something introductory for teaching, this will do.

is this some sort of MIT class to undergrad or general public?

At MIT, the month of January is a reserved time separate from the normal terms called IAP - Independent Activities Period. During IAP, there are many seminars, panels, talks, and short courses (and some long courses taught in an intensive fashion) on a huge variety of topics. Many are taught by students, faculty, staff, and area alumni. While a few classes do give credit, fulfill requirements, and the like, oat exist simply to learn cool, neat, useful, interesting, wacky, etc stuff.

ohh. Only if it was to public, hop on the red line and walk 10 mins.

can't see anything on that projector :-(

Yeah, sadly the screencasting software malfunctioned on some of the sessions. We're thinking of re-doing some of them with a proper microphone and screencast though (see also https://news.ycombinator.com/item?id=19081669), and will certainly announce when/if that happens! For what it's worth, the lecture notes should capture most of the actual contents, and in the same order.

wow this is so cool!!!

Thank you Sir For This Amazing Post.It Really Helped Me To Buy new Tolls For My Industries.

> https://hacker-tools.github.io/lectures/

Literally none of these have anything to do with proper information security/hacking/penetration testing. Will you please consider to stop abusing the term "hacker"?

So, the term "hacker" and "hacking" has traditionally meant someone who is exploring the limits of systems (whether computer-based on otherwise) in a playful manner, often finding "clever" ways to make the system behave in unintended ways. The term is arguably even broader than that — see the Wikipedia entry on Hacker Culture[1] for more. The use of "hacker" to mean the exploitation of computer systems is more recent, and was traditionally referred to as "cracking". The two terms do have a decent amount of overlap, but the latter is much more commonly associated with malicious intent, whereas hacking really just means that a desire to push systems to their limits and see what kind of cleverness you can pull off!

[1]: https://en.wikipedia.org/wiki/Hacker_culture

They specifically say in the course overview that they are using the term "hacker" in this sense: https://en.wikipedia.org/wiki/Hacker_culture

Not "hacker" in this sense: https://en.wikipedia.org/wiki/Security_hacker

It is you that should consider to stop shoehorning a term with now two common meanings into the one that you prefer. Once upon a time hacker meant nothing of what you are referring to, now it is 'dual use', no need to go overboard in the other direction and forbid the original one. For reference see 'The hacker from MIT'.

- via Hacker News

"Hacker" and "hacking" at MIT have a couple of meanings, neither of which specifically means trying to "hack" into a system. "Hacking" either means 1) Working hard on a problem set or other project or 2) Pulling off a prank on the MIT campus without permission of the administration, e.g. turning the Great Dome into R2D2's head.

So in the context of some people from MIT putting out a list of resources, originally for others at MIT doing #1 above, "hacker" is the right word to use.

Hacking is not limited to security. People use it also to name tinkering and customization-topics.

Forget university, most of these this should be the content of K-12 computing/programming classes. It's simpler and has much more real world value than a lot of the crap the tech, like graphical iPad programming or how to use a specific version of word.

The article on editors is basically "here's the names of 5 editors, now learn vim"

I know this will piss off vim users but in my experience vim users are seriously out date when it comes to dev tools. It's like they never left the 70s. vim might be great, it might be available everywhere but when I see a vim user I then see them use something like gdb and have clearly never experienced a modern debugger because if they had they'd be infuriated at how their debugging tools haven't improved in 30 years.

I'm sure others could go into all the features a modern editor or IDE provide. The vim users will say they don't need that crap. They sound like some grandpa saying "back in my we had to walk to 5 miles to school, no one had invented bicycles or cars or buses. Don't need those new fangled things, now Get off my Lawn!"

My experience of IDE users (myself included much of the time) is that they don't know how limited their debugging experience is, just breakpoints and how to step through code. Things like breaking when a variable is set, debugging from a core dump, creating a core dump under certain conditions, viewing the assembly, setting variables on the fly etc, are features of a good debugger (often the one they're using) that are obscured from them.

If you want that shallow level of debugging you can do it from inside vim though: https://github.com/idanarye/vim-vebugger

Exact opposite of my experience. IDEs let me easily add conditional break points, toggle between source, disassembly, source+disassembly, set variables just by clicking on them, view multiple areas of memory in different tabs/windows. Set how that memory is viewed (bytes, words, dwords, floats, as a type of class/struct, array of classes/structs), show different contexts at once, have multiple windows per thread/process, never seen someone use one that didn't know those features exist as they are all obvious to discover in an IDE.

sounds like you've never used one but then that's what I'd guess from watching most vim users. All of those have been standard features of visual IDE debuggers for over 30 years. Mean while instead of seeing every thing update live I watch the vim users type various gdb print and dump commands at the command line and then watch that data scroll off their terminal instead of being updated live as they progress like an IDE debugger.

> sounds like you've never used one but then that's what I'd guess from watching most vim users

You'd be mistaken, I've spent a huge chunk of my adult life inside Visual Studio. IME most devs don't even know you can add a condition to a breakpoint (hiding feature behind right clicks aren't obvious) and anything complicated turns into a very convoluted process very quickly. Even setting conditional breakpoints are basically an input into the command interface, essentially what the windows run menu is to the command line.

Take this short tutorial (https://amazingdim.wordpress.com/2014/02/01/gdb-script/) on gdb scripting and show me how to do similar through an IDE. With Visual Studio at least it is much more complicated, here are a few google results on that path:

1. https://docs.microsoft.com/en-us/visualstudio/debugger/using... 2. https://blogs.msdn.microsoft.com/debugger/2009/12/30/what-is... 3. https://stackoverflow.com/questions/47513337/howto-debug-a-n... 4. https://docs.microsoft.com/en-us/windows/desktop/wer/collect...

>IME most devs don't even know you can add a condition to a breakpoint (hiding feature behind right clicks aren't obvious)

Can confirm: I sure didn't. ...honestly most of my debugging is with print statements.

You’re assuming that Vim users use only Vim. I use Vim all the time for editing config files, small Ruby scripts, and JavaScript projects without types – for these cases it is among the best tools for the job. And when I work on, for example, big Java projects, I use IntelliJ IDEA plus a Vim keybindings plugin, so I can use the IDE’s debugger and its refactoring support. Don’t make sweeping generalizations about Vim users not knowing when other tools are appropriate.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact