I am liking the kind of articles on this blog but unfortunately. I cannot find an index page that lists every single post every written with a link to it. Could you kindly add such a page or point me to it?
> The same effect applies to passenger planes. Airlines complain that they are losing money because so many flights are nearly empty. At the same time passengers complain that flying is miserable because planes are too full. They could both be right. When a flight is nearly empty, only a few passengers enjoy the extra space. But when a flight is full, many passengers feel the crunch.
But my complaints about flights being too full is based only on my own experience. I don't think people are upset about airplanes crowdedness because they surveyed a bunch of people and concluded from the survey that airplanes are crowded.
> But my complaints about flights being too full is based only on my own experience. I don't think people are upset about airplanes crowdedness because they surveyed a bunch of people and concluded from the survey that airplanes are crowded.
This is still falling prey to the error described in the OP.
Imagine that 99% of all flights were totally empty, and the remaining 1% of flights were completely full. Despite the fact that 99% of all seats are vacant in this scenario, 100% of all flyers will have the experience of being on exclusively packed flights.
Except that with what I described, there is no paradox. If I report experiencing flights that were all full, and someone says “but 99% of flights were empty,” there is nothing surprising or counterintuitive about that (other than that I would have expected airlines to be more economically reasonable). And it certainly is no consolation to me!
Yeah but I think they have a point in that when I book a flight in that scenario—if we don’t require that all the completely empty flights remain completely empty—then I’m going to wind up as the only passenger on one of the (previously) empty flights, rather than ending up on the full one. Unless the empty flights don’t allow bookings.
I think the idea is that by posting about it online you form a small piece of an informal survey we all conduct continuously. There are more people in your position than people who fly on empty planes, so the reasoning still stands.
When 737 Max’s were…having difficulties, Alaska Air offered to reschedule your flight with no penalty. We flew anyway. It was glorious. Many rows with one person. Completely empty rows by the bathrooms. My spouse and I had a three seat row to ourselves. It was flying like it was 1999 again.
(And this was Seattle to Orlando, not some puddle jumper to Bumphuck, Nowhere.)
It really depends on where (and when) you're going. I've had a decent number of partly full flights going to oddball small airports. But the big hub-to-hub flights tend to be nearly if not completely full.
"Inspection paradox" seems like a misnomer. It has nothing to do with inspection and it isn't a paradox. It's just selection bias [1] or more specifically, sampling bias [2]. In the case of class sizes, are you asking the average class size from the point of view of a student (N per class) or a teacher (1 per class)? Take the extreme case where you have one class with 99 students and one class with one student. From the student's point of view, the average class has 98 students. From the teachers' point of view, the average class has 50.
It is called that because one of the most common places it turns up is managers/inspectors doing routine work. Typically if a manager checks in on their report they'll be stuck doing something that takes a lot longer than usual. If an inspector inspects ... I dunno, widgets, then the widget will typically be unusual because of selection biases.
I'd expect that since this is the usual workplace introduction to the economic value of knowing about the bias it got its name there. A lot of confused people trying to work out why their data makes no sense.
The other fun workplace paradox would be if HR ever tries to be data driven, does some metrics over the engineering department and works out that a degree is inversely correlated with any attempted at measure of skill. Fortunately most HR persons are not interested enough in stats to try that approach.
Could you elaborate more on the inverse correlation between degree and skill? Do you mean that usually people who did not go to university actually went straight to work and had the chance to get more skill as opposed to people with a degree that actually started later?
Simpsons paradox. People without qualifications have to be obviously competent to be hired to do a job. If someone is clueless they probably slipped in because they got certified somehow (like with a degree).
Expect a negative correlation between certification and competence (in the workplace) because the workplace only reliably excludes people who are incompetent and unqualified. So the population sampled is made up of [qualified, competent], [unqualified, competent] and [qualified, incompetent]. And anyone who isn't ready for that will get very confused when they try to work out how much value a degree adds in their pool of programmers. Or any department, really.
That makes sense, but I would expect this paradox to vanish (or at least get weaker) as you go higher in the hierarchy of technical positions (i.e. from junior to lead, to senior, to principal etc.). I would expect the workplace to somehow naturally get rid of the incompetent people, so that after a certain point you're only left with [qualified, competent] and [unqualified, competent]
One of the best ways to get rid of someone is to recommend them highly to an open position somewhere else. Sometimes a higher level position, or management.
When a software engineer gets promoted to a senior role, their responsibility changes to impact a broader timescale. It's entirely possible that promotion is the very thing that masks their incompetence.
For example, a junior developer is expected to manage implementation details, while a senior developer is expected to manage business logic. Incompetently designed business logic is noticed later, and can often be blamed on trivial implementation failure.
A special case of selection bias is sampling bias (as you said with "specifically"), and the inspection paradox is a special case of selection bias — it is specifically about whether you “inspect” a member of a population or the population as a whole. The article you linked on sampling bias has a list of “types” — https://en.wikipedia.org/w/index.php?title=Sampling_bias&old... — and most of them don't fall under the category of inspection bias/paradox (while the example with class sizes does).
So it's actually useful to have this article that collects many examples of this specific kind of sampling bias (and specific kind of selection bias). I especially like the one on relative speeds:
> when I overtook slower runners, they were usually much slower; and when faster runners passed me, they were usually much faster.
Just because something isn't a logical paradox doesn't mean it shouldn't be allowed to be called a paradox...it can also just mean "apparently self-contradictory"
If you don't have 40 minutes, just pause the video at 15 seconds in and read the screen, you'll get the gist of it. This one is category 3: "counterintuitive fact" or "veridical paradox".
Selection bias is something slightly different. This isn't a bias.
This is a misleading conclusion. The data is correct, but the very act of inspecting that data leads to a confounding result.
I think the work "paradox" is imprecise, but it does fit the spirit of the problem well. A layperson may expect that data will draw a useful conclusion. The fact it does not feels paradoxical.
No, there is nothing misleading about it. There are two equally valid conclusions. It depends entirely on what you take to be a data point. If you are asking about average number of students per class, and you have 100 students and 2 classes, do you have 100 data points to consider or 2?
reply