As an aside, there is a typo in the parent's link:
Well, Chebyshev inequality always applies. It just might not be the tightest bound possible. Unless we know the distribution better, this is the best estimate (tightest bound that's provably correct given the single assumption about standard deviation)
Since a whole bunch of things are happening at once, all shooting particles at your detectors, you get a whole mess of particles all at once, and it's kind of a pain to separate which bunches of particles go together. So what they're saying is that the detectors have been seeing bunches of particles that look like they might have come from decaying Higgs Bosons, but there's still some chance that some other bunch of particles could have have been decaying in just the right places to look like decaying Higgs Bosons.
Isn't it usually accepted that it is statistically significant when p goes above 0.95? If so, why aren't they happy with a 20 times better result?
Consider a situation where they test 1000 hypotheses, only 100 of which are actually true. Of the 900 false hypotheses, 9 will, by chance, achieve a 3 sigma "99% chance of being right" result. And hence something like 8.3% of the hypotheses tested as "true" will be false positives -- not 1%.
Physics wants a much greater degree of certainty, and since they have the ability to get it they insist on it.
This http://xkcd.com/882/ sums it up pretty well.
You have to make sure to have results that are far far better than the number of experiments you are running. Otherwise you are virtually guaranteed to find some result that will seem right, but isn't.
(For example a 1 in a million occurrence (per person per day) [would] happen about 7000 times every day.)
Firstly, it is a lot harder to repeat experiments in biology than in particle physics, as (as far as we know), all electrons are the same.
Secondly, biologists will, in general, not make bold claims. A paper "a possible link between X and Z" that works with p>0.95 and states that further research is needed is not a lie; the popular press makes it a lie by changing it to "OMG: X CAUSES Y".
Just because someone uses weasel words ("possible") doesn't change the end result: It's a wrong result.
I'm not blaming them - I understand better results are not possible. But it doesn't change the fact that a tremendous number of results are wrong.
It doesn't help that they often search for very subtle results. "It helps, but only a little." It also doesn't help that everyone responds differently to things. It makes the research very hard.
Anyway, I was just explaining why p95 is not accepted anywhere else except biology - biology just doesn't have any other choice. They don't prefer such low results.
I generally treat particle physics as a black box from which magic appears, but my understanding was that 3 is where things start getting interesting, and 5 is where it's considered pretty conclusive, so wouldn't going from 3.5 to 4.3 be a significant jump?
I would be interested to learn about common procedures from physics as well.
I guess Wikipedia can give you an idea: http://en.wikipedia.org/wiki/ATLAS_experiment
A common statistical method used in particle physics is the Monte Carlo method: http://en.wikipedia.org/wiki/Monte_Carlo_method
It might be like expressing amazement that someone in the world won the lottery. There's a difference between someone in particular winning the lottery, and anyone winning the lottery.
Xkcd says it better than me. (hat tip to starwed)
Then after the party, everybody has the hangover that may never end, because finding the Higgs right where we expected it is potentially the worst case scenario: http://www.sciencemag.org/content/315/5819/1657.full
There are a ton of questions that Higgs cannot answer already and yet if we find Higgs precisely, research comes to a stop?
It almost feels like there's a split between theoretical physcists and everyone else, with theoreticians saying if you take away our broken toy, you better replace it with something we can play with!
Given Nature already trumped Einstein, his contemporaries, and all since, I think we can be fairly secure that we will ALWAYS having find something new to learn or discover from practical science.
After all, that's how we used to almost exclusively learn before scientists went a bit crazy with maths. These days there seem to be many more models out there than there is good science behind it - at least to a layperson like me. :)
This argument should also be read along with the fact that last I knew, none of the accelerators have been able to turn up anything else particularly interesting either. Some of the supersymmetry theories predicted particles in ranges that we should be able to see (barely) and none of them have appeared. We're down to hoping that there's something else to find in the extra room the LHC will give us at full blast or we really will be up a creek.
"These days there seem to be many more models out there than there is good science behind it - at least to a layperson like me."
And in fact your observation is connected; particle physics has been starved for data and in the interim have come up with all kinds of things, trying to find things that may have testable consequences. This would go a lot better with some data.
I know of only one, and whether it tests anything at all depends on which interpretation of QM you have. Based on a Geiger counter, either place, or don't place a heavy weight. Try to measure its gravitational pull regardless of what you do. It only measures a pull if you placed the object. If you believe in the Everett interpretation, this says that gravity, at least to a first order approximation, splits with the universe. We do not have sensitive enough instruments to measure non-linear differences from GR.
History tells us that theoretical science done in the absence of experiment is unlikely to lead to useful knowledge.
(especially question 7) will help, even though it does not give the name for such an experiment.
So, whether there's science still to be done and whether we'll ever be capable of building apparatus that can actually test it are two different questions. For instance, what if the next interesting thing post-Higgs Boson happens at energies 1000 times bigger? There's a good chance we'll never build an accelerator that powerful.
It is possible that we will discover new phenomena and new ways to test gravitational theories once we can observe gravitational waves. I expect we will detect gravitational waves in ~10 years and identify a specific source in ~20 years; sooner, if there are some powerful sources that we did not think of yet.