And, like you say, it does create a very different attitude toward error. If you're a politician, admitting you're wrong is a weakness, but if you're an engineer, you essentially want to be wrong half the time. If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.
That's idealism talking, but practically speaking experimenters are invested in the outcomes of all but the least interesting experiments. Nobody wants the search feature they spent a month implementing to come back with negative results on evaluation. This is not only true at Google.
Some of the stuff in this sounds nice but I feel it lacks some honesty. "What is the thing you have been the most wrong about?" And you say that the worst error in judgment you've made is an off-hand remark in a meeting? Let's be serious here.
His call on big mistakes from Google seems good but I doubt Google could have executed on it even if they had known how big Facebook would be beforehand. Big G is not a product company and is best at making products that minimize the time you have to spend using them.
"Some of the stuff in this sounds nice but I feel it lacks some honesty. "What is the thing you have been the most wrong about?" And you say that the worst error in judgment you've made is an off-hand remark in a meeting?"
Huh? That isn't how I read it at all.
From the article:
"Question: What about you yourself—what have you personally been most wrong about?
Answer: One thing is how fast things change.[Stuff about meeting here]You think you have this experience—and we talked about how important experience is for having intuitions—but experience can go out of date very quickly."
To me he seems to be saying that one of his major mistakes is underestimating how fast things change, and underestimating the erosion of the value of his experience in guiding his intuition (He just finished talking about the importance of intuition for situations where statistics and data can't help/don't exist.)
Given that he was speaking about a professional context [1] and the position he occupies at Google (Director of Research), where anticipating change would be presumably part of his job description, that sounded to me like admission of a major lapse, somewhat different from "My biggest mistake was an off hand remark I made at a meeting", as you seemed to imply.
He was using the meeting incident as a concrete example of a rather large failing. How is it "dishonest"?
[1] The whole interview is about errors in a professional context , about Google and his work there, so I wouldn't expect to hear about whether he got stoned as a teenager and got thrown in jail! ;-)
I think it was more of a regret that his intuition about the problem area gets invalidated so quickly. The top technical guys are supposed to have good intuitions about what works and what doesn't; this is a major part of the value they can add. So I take this as a very serious self-criticism.
"is best at making products that minimize the time you have to spend using them"
An excellent observation - I use Google a lot the total amount of time I spend on their sites really is pretty small and the "brand loyalty" for me is almost non existent.
This reminded me of information theory. The idea of entropy in information - that if you can predict the next 0 or 1 with better than 50% certainty, you can compress that information (or in this case, you could have designed a better sequence of experiments).
I do think it's worth differentiating between types of failure, however. You don't want 50% of your experiments failing in the sense of failing to give you an answer in either direction (i.e. being inconclusive).
And this leaves aside the topic (covered in the article) of the more serious kinds of failure - of live code etc. You probably don't want that happening 50% of the time.
I had a boss who always pushed me away from low-probability experiments. Whenever I suggested trying something that was unlikely to work, he probed to see if the single low-probability experiment could be restructured as a series of 50/50 experiments. It turned a lot of failures into partial successes that could be applied to future problems.
That's precisely where Norvig is coming from. Given their infrastructure, it's entirely possible that some portion of their A/B testing is run this way (i.e., test thresholds chosen to maximize information gain).
It depends on whether you're a production engineer making things for use (ICs, software, bridges) or a development engineer trying to push the limits of what can be done.
And, like you say, it does create a very different attitude toward error. If you're a politician, admitting you're wrong is a weakness, but if you're an engineer, you essentially want to be wrong half the time. If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.