As a sidenote, isn't this pretty much why the PDF remains so popular as a format, at least in science - you get a typeset copy with no worry you'll ever lose access or not be able to read it... you just need that spare hour or two... :)
Does anyone have an application like that?
Any suggestions on what authoring tool(s) to use to write a book like this that has lots of matrices, diagrams, and notation?
As for your question, usual answer would be LaTeX + (maybe) some specific plugins (like TikZ for diagrams, or tufte-latex for specific layout, or whatever), but due to some irregularities here I suspect that this particular book is actually hand-made in something like Microsoft Office. I well might be wrong, I'm not sure, it just seems like that.