I once suggested an alternative to research depts. Instead have a group that builds tools for the people building products. That way you still get to be one step removed from random customer needs, but you're also helping the guys who do face them.
I remember that essay, and my employer works a little like that (we have a platform that all of our products are built upon). IMHO, it doesn't work quite as well as it sounds like it should.
Problem is, information is lost every time someone communicates their needs. Customers don't actually need what they say they need; there's always some detail that they forgot to mention that only shows up when you watch them use the product. That usually changes the tools that the programmers need, which often aren't the tools the programmers say they need, because programmers have their own preconceptions about how the solution should be implemented. Someone who's only done JavaEE programming will likely say "I need a tool that automates maintanance of my Hibernate and Spring configuration files", when they really mean "I need to store my user's session data between sessions."
There's often a strong organizational pressure to use the output of the toolmakers, even if the tools aren't suitable for the job. In my case, I'm not sure that I'd use our platform if just given our customer's requirements and my choice of toolset. But I can't really say that, because it harms morale and makes me seem like a lone wolf who can't work with other people. (Okay, I have said that, with the expected result.)
The nice thing about research labs is that nobody expects them to be useful. Other programmers can pick up or drop their work based on whether it actually is useful, not by some corporate policy. Unfortunately, the people most likely to pick up an interesting research lab project generally refuse to work for a big company. So there's a long history of guys in garages ripping off the best ideas of corporate research departments, and eating the parent company in the process.
If I were managing a large, cash-rich corporation, I'd have a small exclave a mile or so away from the main campus. It would have two buildings: a top-notch research lab, and an incubator (and a shared cafeteria, of course). The research lab would be staffed with high-priced Ph.Ds and tenured professors, the best in their fields. The incubator would be full of high school dropouts on grad student salaries, but with $40M+ stock grants contingent upon building something that millions of people will use. All executives from the parent company would be prohibited from setting foot on this campus.
Come to think of it, this sounds a lot like grad school, but without the bullshit.
R&D labs fail at innovation because they're not designed to innovate. They're designed to invent. And they do a pretty good job at it as well.
Without R&D labs we wouldn't have digital cameras, laser printers, the GUI, the mouse, cell phones, pharmaceutical drugs, etc.
If you want a good book on the subject, grab "Open Innovation" by Chesborough. It does a good job explaining why Xerox PARC wasn't able to monetize many of its inventions even though its managers were highly competent leaders who were following the best practices of the time.
I would actually say they don't fail at it, in much the same way that dolphins don't fail at flying -- they aren't even trying.
After the researchers at Xerox PARC invented, they innovated by splitting off into different start-ups to innovate, i.e. introduce the new product that had already been invented.
