The book Peopleware backs up the failure of open plan offices with references and data. For any kind of knowledge work, distraction is kryptonite.
I really enjoyed the book, and the info about open plan offices was really interesting. I was previously a fan of the open plan, but now I think if I ever design another office, it would be common areas and offices that close.
We had this setup at Apple (early Infinite Loop days) and it rocked. The best environment I've ever done engineering in, period.
Being able to shut your door for a few hours to crank stuff out is incredibly important. Being able to walk a few feet and collaborate is also incredibly important. They're not at odds.
It doesn't take rocket science to design a space that works. It takes something harder: Money, and the ability for non-engineers to listen to people who haven't been drinking the latest silver bullet kool-aid.
"People can't concentrate in noisy environments." How hard is that to get?
Peopleware has some great data about how listening to music can lower distractions but also reduces problem solving ability. The upshot was that test subjects could plow through code while listening to music, but tended to miss optimizations and shortcuts.
I think Peopleware is a great book, but it makes bold claims with very little experimental data. I personally find this hurts the integrity of the book and would have preferred if the authors were a bit more humble.
The music vs silence point, for example, is based on a tiny experiment with a small sample size. The kind of conclusions that are drawn from it in the book are a stretch.