The reason is well understood, but not by mainstream economists. I suggest reading about Hyman Minsky and Steve Keen's research. To summarize, increases in total debt (private or public) contribute to demand, while decreases in total debt subtract from demand. This results in a financial instability. More demand => more debt-fueled investment => more demand. Eventually, businesses that could never have survived at the bottom of the cycle can get huge loans; they might not even seem speculative, because demand is so high and keeps growing. However total debt rises too quickly, due to speculation and Ponzi financiers, and it can't rise at such rates indefinitely. Once debt growth begins to slow, businesses financed primarily by debt begin to fail, because demand slows and they can't convince anyone to give them more loans to cover interest. This triggers runaway deleveraging and fall in demand. In such a down-turn, even businesses that could have survived at the bottom of the cycle have trouble because they've taken out loans or sold shares to invest in growth, since they expected demand to continue to rise. Instead, demand shrinks, because much of it was debt-fueled and unsustainable.
What is required is a good theoretical explanation for why investors fail to predict the point they are in the bubble and rein in speculation, since obviously this cycle has happened many times in the past.
I think the problem is human nature. People forget. Sure, this has happened many times before, but the most recent time was, say, eight years ago. That's a long time in peoples' memories.
And, while there may be this nagging memory of trouble, right now there's money to be made...
If you want a theory for _why_ the future is unpredictable, read up on chaos theory.