My personal experience* is that some of that failure rate can attributed to students not understanding what a PhD is for. It is a qualification to conduct research... it is not about creating something cool/interesting and/or useful (particularly not the last one). It's about the journey not the product.
Unless you intend to stay in academia, the qualification probably won't pay off in terms of a higher salary vs. the years you wasted. And if you do intend to be in academia, be prepared for a long (possibly indefinite) wait before you get a permanent lecturing/research position. Post-doc life is not easy.
* 3 years wasted working on a PhD, no thesis submitted in the end, walked away back to the real world.
> It is a qualification to conduct research... it is not about creating something cool/interesting and/or useful (particularly not the last one). It's about the journey not the product.
Yes. Yes, yes, yes. New PhD students should have this tattooed on their arms as part of the admissions process. I wasted an entire year trying to come up with a dissertation project because I didn't understand this fact.
I think the real danger with a PhD program is that it is so easy to wander into a program, especially once you've gotten a master's degree. It feels like the right move and the next logical step, just like the previous degrees were. It's almost painless to stick around, look into interesting topics, get paid a livable salary, do some research and attend a few conferences. It can even feel like real meaningful work, without the indignity of having to find and start a job.
There is this kind of background worry that the whole thing is getting nowhere productive, or at least not at all where it's supposed to go, but it takes a few years to build up. That ends up blossoming into the regret of three years wasted, instead of a few stressful months (if that) in job limbo.
It's not all bad though. My curious distraction from my 'real work' was learning to program. I ended up with an employable skill that I enjoy using and building on. Still, though, 3 years.