As someone who just graduated with a B.S. in chemical engineering and will start work toward my PhD in the fall (also in chemical engineering, but will actually involve a TON of programming), this strikes a chord with me.
A few takeaways from the article:
-The author does not give himself nearly enough credit. "Self-deprecating" is probably the right term. He considers himself average, but he is far above average. I mean this as a compliment of course.
-I think some of those tools he wrote could actually help me with my work (particularly CDE) -- I'll have to go take a look.
-I am going to have to learn how to focus, and very quickly. I consider myself very motivated/stubborn but I get distracted so easily that it's ridiculous, and this is not going to work for grad school unless I alter my behavior substantially before I get there.
-Before reading his book, I too had the idealized notion of "brilliant idea, years of writing one paper about it". The reality isn't quite as fantastic, but I appreciate learning the truth over belated disillusionment.
-Just realized the author is pgbovine -- excellent work! I currently have a completely free summer to do whatever -- would you have any tips on how I should prepare before I get to grad school? I already know which professor I will be working with, so I'm trying to figure out what level of "initiative" I should take before I arrive.
I don't know whether that's the convention in your field (or with your advisor), so I don't want to make any generalizations. It's certainly possible to graduate with one big idea -> single paper, but that's just not what I ended up doing.
re: focus -- yes, that's key. but of course, it's bad to be blindly focused down a bad direction, too. don't put too much pressure on yourself early on, since nearly everyone flops and flounders.
re: what to do over the summer -- i don't know if there's much you can do to prepare :) maybe reading your professor's past papers? again, i wouldn't stress about it too much.
I read this in its entirety, unable to go to bed before finishing. This is amazing. I'll admit that I started out with skepticism towards Philip's initial research goals.
Instead what I found is an application of immense talent , culminating in software projects that are nothing short of amazing. For what it's worth, something like CDE seems like a holy grail in many scenarios. I've spent some time time thinking about distibuted cluster management software (Google's borg, MS auto-pilot, Hadoop YARN) and the problems these system tend to run into when encountering stateful services. Naturally, I'll spend a bit of time reading up CDE after this.
As Xcelerate said it's quite obvious that the author is _nowhere_ close to average, even in a top-tier program like Stanford. I also admire is that the author was able to maintain this healthy degree of self-deprecation while still doing the inherently (to a research or developer) distasteful but vital work of selling.
Many of my team mates at my current and previous jobs hold PhD. I come from an academic family: my dad was a full professor (e.g., habitulation et al) in our home country, my mom completed her PhD dissertation but wasn't allowed to defend for reasons best answered by a Google search. However, this writeup still shocked me in many ways (the sheer grit and tenacity required on top of extreme mental horsepower). Thank you for sharing this.
Edit: one _tiny_ suggestion -- pgbovine, I like how you've planned for the book to be accessible to those outside of computer science/software engineering fields (e.g., explaining what device drivers, Python, et al were). That said, I still think there are a few pieces that could be improved in terms of the paper's accessibility to non-CS folks: going into more details into potential sources of bugs (to explain the motivation behind your initial research), the full pain of systems' environment management, etc... You should definitely publish this for a wide audience (at the very least gearing this for students or professionals in all natural sciences and other quantitative fields).
> it's quite obvious that the author is _nowhere_ close to average
Most grad students are exhilarated to work with a brilliant researcher but it can crushing when one learns his or her own limitations.
Like the author says, probably 1 in 100 are the equal of their advisor and capable of becoming a professor at a comparable university.
I wanted to add a little about the end-game. For me, the decision of what to do next followed a slightly different path. In the fall of my 5th year, while planning to graduate, I had to really think about what to do next. Academia sounded interesting, so I went ahead and applied to tenure track jobs, and did interviews. Come winter, I had an offer from an Ivy League school. But I had also wanted to do startups for a while.
From my experience, working on projects that were not provided/sanctioned by advisors was somewhat like working on a startup. You have to build things that others find useful/interesting, sell the crap out of it in publications, presentations etc, and I also had to fund it.(my research was building systems software that improves user experience in poor network environments, like those common in developing regions. One of the definite perks was travelling lots and deploying stuff I built.)
In other words, I have no idea why most PhDs are not be expected to go work on a startup when they finish. They are domain experts, and are expected to be good communicators when it is all said and done.
So, when it came time to make the call---an assistant professor, or a startup founder---I chose the later (and disappointed my advisor). Some friends tell me I could have done that without a PhD, and it was a waste of time. I think a PhD naturally leads to a start up. The Grind, as the op calls it, is something that teaches you a lot about yourself, and surrounds you with some of the smartest people you'll find. Your job, every single day, is to innovate [with a more flexible runway]...and you can make (and learn) a lot out of it.
Knowing what goes into a Ph.D. is invaluable in making such a decision, and this book was quite helpful in that regard. Thank you!
maybe i didn't make the disclaimer big enough, though :)
Please don't take my comment as "you ruined my motivation for further education"; it was meant as "you saved me the trouble of finding out the hard way that this really wasn't what I wanted".
I have never been interested in being a professor, so I would be in it entirely to push my limits. But if I know I won't enjoy it (and as far as Ph.D. programs and publication go hand-in-hand, I know I wouldn't), then is it really worthwhile? That isn't to say I wouldn't emerge a stronger person at the end, but I would rather find ways to push myself intellectually that I will enjoy. I believe they're out there.
Edit: I just read a post about your book at http://blog.regehr.org/archives/743 which you might be interested in.
I really enjoyed my PhD experience; I had time to explore who I was and was able to accomplish what I wanted to. It wasn't so much of as a useless grind as it was a journey to enlightment, but my experience is unique.
All trades have their harsh, distasteful parts. The rest of the world is mostly caught up in a rat race about money, artists struggle to get exhibited in galleries, etc. You shouldn't focus on the negative parts, the decision should depend on what your ambitions really are.
What do you want to do when you "grow up"? If you are hanging out on HN, I assume you have some interest in start ups...read my other comment. And I can give you a lot more reasons why you should continue [especially if you are at one of the better institutions].
The best thing to do if you are facing the option of a PhD and also interested in other paths (startup, bigco, etc) is to fully understand what the PhD will do for you and what it is like. Including the impact it might have on your mental health, career, and ability to do a startup later. Reading essays like pgbovine's are a great way to fully understand this decision.
re: distilling -- my main intention in writing this memoir wasn't to provide advice, since it's hard for me to determine how much facets of my experience generalize. i do have plans to convert the "lessons learned" section from the epilogue into a talk, though.
i have this other article from the beginning of my third year that's more of the "advice" type, although again, I'm wary of making generalizations about Ph.D. life:
(other fun fact - i used one of your blog quotes as the title slide of my Ph.D. oral defense)
I just read this a few days ago (was actually recommended here on HN ) and thought it was terrific.
That might be one of my biggest problems in science - I have a hard time memorizing people's names.
here's an HTML version too ...
A couple of childhood friends just finished their third year of their PhD programs, one in molecular biology and the other in physical chemistry, and are very much in that all-too-common third/fourth year rut. I wish they'd had this to read three years ago!
I actually chuckled during some of those early chapters... "It takes 4 weeks to write a good paper." Followed later by, "We ended up submitting an embarrassing jumble of text filled with typos, nonsensical sentence fragments, graphics without any explanations, and no concluding paragraphs. It was a horrid mess." It's even funnier when something like that gets accepted (or the really polished one gets rejected!).
I plan to send this to some of the more junior grad students I know!
I heard you'd graduated, so I wandered over to your site today. Didn't realize you'd put up this article so recently.
I am sitting at an airport on my way back from a top-tier conference and the Epilogue on Page 99 resonates so much with my own experiences. Thanks a lot for writing this :)
I'm only halfway through this, but everything is true, especially about research agendas and insider academic sub-communities.
While you are at google, you might consider continuing to engage with the academic community to a limited extent. Onward! is very different with a new/improved community that is interested in bridging gaps between PL/HCI/systems and isn't so concerned about the publishing rat race.
re: Onward! yes, i've heard good things about it; will keep a look-out.
No, there was autopackage, alien and any numerous other programs and battles/discussion long email threads that worked not just for python but for any language. Technically speaking as I see CDE, it is useful for python and standard installs in very select cases. For large research projects where the variables and runtimes will differ with no rhyme or reason other than "That's what we started with". It's probably a lot more of an issue to get running properly.
I'm sure there is no easy answer, it's really a human problem. What would really be highly useful is for standardization on a linux distro specifically for research but even then that would only be useful for at most an organization/institution/research lab. Getting researchers to stick to such a distro as a general rule is impossible without weapons.
Especially early on, I was so personally attached to my work that any problem (eg, an upper bound not working out right) was devastating. But then, when it did work, ecstasy. And then, uncovering bad algebra in the derivation caused despair. And then, error corrected, euphoria. Round and round.
Are those people actually rare in academia?
What I meant was that in the context of a highly competitive research environment those who were succesful had learned that to succeed they couldn't afford to focus on anything that wouldn't progress their own careers - and that meant (and presumably still means) publications in the rights places which are usually completely uninterested in practical details and focused more on pure mathematical/conceptual concepts.
I'm not a systems researcher but I work in a systems research group (at MSRA) as a PL researcher, which is a very different community; we tend to be more designy.
It's not that they don't look for things with practical results. It's just that there's no incentive to continue beyond the point where you demonstrate the contribution.
Having recently left my program (but enjoying research), I've actually been spending a huge amount of time planning out and slowly executing building a better tool chain for data analysis, and one of the really exciting details that has popped out of my design is a very nIce support for incremental computation and computational provenance, while having all of the constituent parts be libraries (rather than needing to mod a run time)
Point being I was really intrigued to see the thinking behind other folks getting drawn into that space
I'm actually right now figuring out if I can align the stars so I can just support myself while working on this tool chain project of mine, but we really should chat in more detail at some point! (lets one of us get around to emailing the other in the near future)
Off topic, but I chuckled as I'm interning at Microsoft this summer and actually came across a paper on Klee this last week as I was doing research to guide my own testing project. I did a double-take when I saw the name.
Another thought I had is whether you could implement this in pure Python using the introspection features of the language. Finally I was reminded how with certain languages such as Lisp, there is this notion of an "image", which is a dump of the state of the interpreter. Does that achieve the same end as this? If so, wouldn't that be a much more general solution?
Thanks for writing this.
I'm glad I'm not an academic
Just wanted to offer the counter opinion. Each to his own...