I've noticed a profound shift in the past decade away from corporate research labs such as IBM Labs and HP Labs where they worked on medium-term projects developing research prototypes that were sometimes passed onto product teams. In their place, companies such as Google have pioneered a different model of research (https://research.google/pubs/pub38149/), where PhDs are hired as software engineers who solve research problems and write production code, focusing more on shipping production code rather than writing papers (although there have been many great papers that have come out of Google, most notably the MapReduce and Spanner papers). I'm noticing that the vast majority of my PhD-holding friends in computer science are hired as software engineers rather than as researchers. The ones who started out at places like IBM Labs or HP Labs with the titles "Members of Technical Staff" would often end up taking positions at other companies as software engineers.
This development may be fine for researchers who want to work on production code and who don't mind de-emphasizing publishing in exchange of product development. However, what about researchers who want to focus on solving research problems that cannot immediately be applied to products? I'm finding that there's decreasing room for these types of researchers in this economy. More companies have a short-term mindset these days, partly due to changes in management style (e.g., the rise of Carly Fiorina-style CEOs), but also due to the fact that the computer industry has shown repeatedly that large 800-pound gorillas can be taken down by smaller companies. "Why invest in long-term research and long-term planning if there is no guarantee of a long-term future" is the logic of many companies, big and small. The alternative to industry is academia, but there are only so many professorships available, and for those professors, there is only so much NSF grant money to go around, which is highly competitive to earn. Professors at research universities spend a lot of time fundraising; it costs a lot of money building and maintaining a lab that is resourceful enough to perform the research and publish the results necessary to gain tenure.
Short of a major cultural change where companies are encouraged to invest in research at the levels that Xerox and AT&T did back in the 1970s and where we see an expansion of academia similar to the post-WWII boom (which is unlikely in the United States), the future I see for those wanting to work on problems that don't lead to immediate productization is independent research done on a researcher's spare time when not being engaged in "money-making" activity. After all, Einstein did brilliant work while he was a patent examiner, and Yitang Zhang did amazing research while being employed as an untenured lecturer. I would advise today's computer science PhD students of this current reality of research employment. If one wants to work on self-directed research projects, then that person must be willing to have a self-funded research career; all researchers need to be concerned with funding whether that funding comes in the form of a direct salary, a grant, or indirectly through the salary of an unrelated job.
The challenge I'm talking about is papers, specifically the game of publishing. I feel like PhD students spend more time in optimizing for publications than doing actual research. There's an obsession with the number of papers so everyone is trying to eek out a paper for every semi-failed experiment, overfitted model, or unfinished prototype. No one wants to throw away effort on something that basically failed, so they're trying to find the perfect narrative, frame their results just so it looks like they're good, or slice and combine it into something submittable. Every deadline is worth submitting to, the number of selective conferences is growing, and there's incentive to "get on" another paper as a co-author (which means building collaborations, helping out, editing).
For each paper, there's months spent writing, giving and receiving feedback, making figures and formatting. Each submission usually requires some change to the format and language, so upon rejection, the paper is edited and targeted towards the next conference. Then there's the submission game of proposing reviewers, choosing the right track or subcommittee, interpreting reviews, writing multi-page rebuttals, editing and getting feedback from co-authors about the rebuttals, and in the best case a month later, preparing the camera-ready version, the back-and-forth with the publisher, and finally preparing and practicing the conference presentation.
So before great research can truly come out of universities, I think publications need to be deemphasized. This could be a simple norm like judging researchers on their best 3 papers for faculty hiring and research awards. In turn, that would reduce paper submissions, increase paper acceptance rates, and finally -- leave more time for actual research at universities.