A very good essay on Bayesian probability theory and its applicability to rationality is "A Technical Explanation of Technical Explanation" at http://yudkowsky.net/bayes/technical.html . The beginning is more mathy than the latter half, which is more interesting, so don't let the math keep you from reading it and getting to the more interesting later parts. There isn't much math, and the math that is there is pretty simple. If you haven't been introduced to Bayesian probability, you need to at least skim through the article entitled "An Intuitive Explanation of Bayesian Reasoning" that is linked to from the above URL.
If you decide that Bayesian rationality is totally awesome, read the book "Probability Theory: The Logic of Science" by E. T. Jaynes, which is specifically about Bayesian probability theory and its applications, derived entirely from a list of simple informal "desiderata" that we would expect from a theory of probability. He shows that Bayesian probability theory is the only way to satisfy these simple logical desiderata, and that Bayesian probability theory is the unique consistent and natural extension of Aristotolean logic (True/False logic) to real values.
Bayesian probability is a good topic to be familiar with as a general conceptual tool for evaluating information; it's not just some obscure theorem used by statisticians.
ced - "Bayesian Theory" is more than just a formula, it's a way of thinking. A beautiful book exists on this subject, "Information Theory, Inference and Learning Algorithms" by David Mackay, which you can read for free online. Granted, the nomenclature is confusing, there is a Bayes formula. [Edit: jey mentioned Jaynes' book. Another excellent reference. Link to chapter in Mackay (http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/22.40.pdf read from bottom of page 25 on)]
In regards to the unsophisticated aspect of startups, I think that much of the low-lying fruit has been picked. Statistical sophistication is but one ladder to the higher fruits. Why? Social websites have enormous databases of information about their users. People who /understand/ data will be able to take social networks to the next level.
Good cites. I second them. Mackay and Jaynes are excellent but can get out of hand quickly. Duda, Hart and Stork is another good hardcore option. When I need the book "for dummies," which is pretty often, the Weka book and Mitchell's Machine Learning are helpful.
I think it's strange that "Bayesian Theory" has become a concept. It's just one of many formulas in probability theory. Yeah, it's particularly useful in many cases, but learning Bayes theorem without learning the rest is silly.
Regardless, I agree that investing in probability/statistics is a good idea. Right now, there's a rush to fill all the niches of the web ecosystem. Startups can afford to be relatively unsophisticated, because so is everyone. I suspect that eventually, we'll see radically more "evolved" systems displacing the old ones, like Google did to Yahoo. I don't think that those could be built in 3-months, though. I'm curious to see if YC will have to change its model, but that might be a way off.
I certainly see how a Bayesian theory is useful, but I can't fathom how to use it instead of an if statement. What does that even mean? What would it look like? Can anyone enlighten me?
I read the "very senior Microsoft developer"'s point as something like: because if statements are so fundamental, most lowly MS programmers think in terms of them. Google, hires programmers who've taken a course in machine learning or something, and use statistical inference (a bunch of if statements, just with some abstraction) instead of lookup tables (a bunch of if statements.)
Yawn. This is just Google PR speak. The quotation doesn't really mean if statements will go away, it just means that you need to think at a "higher level." This is just someone saying "X is smarter than Y."
If you decide that Bayesian rationality is totally awesome, read the book "Probability Theory: The Logic of Science" by E. T. Jaynes, which is specifically about Bayesian probability theory and its applications, derived entirely from a list of simple informal "desiderata" that we would expect from a theory of probability. He shows that Bayesian probability theory is the only way to satisfy these simple logical desiderata, and that Bayesian probability theory is the unique consistent and natural extension of Aristotolean logic (True/False logic) to real values.
Bayesian probability is a good topic to be familiar with as a general conceptual tool for evaluating information; it's not just some obscure theorem used by statisticians.