I'll try my best to give my two cents (as someone who has done medical school and works in biomedical research). Your explanations all have some merit.
Having spent time working in/with the research and clinical trial divisions of large pharmas (and their oncology programs), I don't think there is an actual "conspiracy". However, there are certainly circumstances where incentives align (or misalign) such that the pharma/cancer industry devotes resources/time/energy into relatively futile efforts. As brought up in this article, currently we see a proliferation of cancer therapies that have slim (or debatable) benefit in progression-free survival or overall survival. Crucially, because CMS & insurers will usually reimburse the exorbitant prices for these therapies, the pharmas continue to search for, research, develop, and market them. Simultaneously, pharmas will continue to point to the high costs of R&D to justify the costs of therapy. (In some cases those R&D costs are quite exaggerated, but that is another story.)
Since pharmas are constantly trying to balance exploration vs exploitation (i.e. a multi-armed bandit), when they find a therapeutic paradigm that is profitable, a big chunk of the research enterprise & industry ramps up to it: NIH grants, university research programs, startups, etc all start pumping out research and therapies and such in that area. In the past decade-ish for oncology, it has been targeted antibodies/inhibitors: all these -mib and -mab drugs. As this article points out, sometimes these targeted therapies truly do work for some sets of patients/diseases. The progress has been undeniable. Sure, there are therapies which give a few months of progression-free survival at the rate of $100k+/year, and then we find that overall survival doesn't budge. But there is significant hope that maybe if we look at the right subset of patients which the right type of disease, or develop a more targeted and more specific inhibitor, we will get even better results.
Which points us to where a lot of oncology is going now, which is toward "precision medicine". Some of the new cancer therapies are useful only if your tumor exhibits a specific genetic mutation. That is, it might not matter if you have lung cancer or breast cancer or colon cancer -- if your cancer has a mutation in XYZ gene, you should be on ABC inhibitor.[1] For some patients, the results for some of these therapies have been near miraculous. For some, it has not. But again, it gives us hope that there is a lot to be learned, as uncovering tumor DNA sequences has not been cheap or easy until relatively recently, especially since tumors exhibit heterogeneity in their DNA. Another hot area is personalized immunotherapy, as with CAR T-cell therapies, which harvest your body's immune cells and custom re-engineer them to attack your cancer -- definitely not a cheap prospect currently.
So as therapies get over the hurdles of showing "enough" benefit vs risk to get through the FDA, and as payers agree to reimburse for them, once again the research enterprise will increase its resources to these new paradigms. And with all that inertia behind it, sometimes the research field is over-excited about the uncovering of an previously unknown mechanistic link and the promise of new therapies... and we end up a little blind to the crappiness of the clinical studies and actual outcomes. I believe the system can be better. But as of now, the costs and the incentives and the burdens of heavily regulated R&D and the sick patients combine to make it a challenging landscape to work in, and alas that is currently how it works.
[1] One example is microsatellite instability / mismatch repair & a therapy called pembrolizumab, which was uncovered during a trial of patients with advanced colon cancer that was going very poorly, except for one patient who had an incredible response to the drug. Further investigation and subsequent studies demonstrated a link, reported in NEJM (http://www.nejm.org/doi/full/10.1056/NEJMoa1500596) and elsewhere, leading to eventual FDA approval for that indication.
Having spent time working in/with the research and clinical trial divisions of large pharmas (and their oncology programs), I don't think there is an actual "conspiracy". However, there are certainly circumstances where incentives align (or misalign) such that the pharma/cancer industry devotes resources/time/energy into relatively futile efforts. As brought up in this article, currently we see a proliferation of cancer therapies that have slim (or debatable) benefit in progression-free survival or overall survival. Crucially, because CMS & insurers will usually reimburse the exorbitant prices for these therapies, the pharmas continue to search for, research, develop, and market them. Simultaneously, pharmas will continue to point to the high costs of R&D to justify the costs of therapy. (In some cases those R&D costs are quite exaggerated, but that is another story.)
Since pharmas are constantly trying to balance exploration vs exploitation (i.e. a multi-armed bandit), when they find a therapeutic paradigm that is profitable, a big chunk of the research enterprise & industry ramps up to it: NIH grants, university research programs, startups, etc all start pumping out research and therapies and such in that area. In the past decade-ish for oncology, it has been targeted antibodies/inhibitors: all these -mib and -mab drugs. As this article points out, sometimes these targeted therapies truly do work for some sets of patients/diseases. The progress has been undeniable. Sure, there are therapies which give a few months of progression-free survival at the rate of $100k+/year, and then we find that overall survival doesn't budge. But there is significant hope that maybe if we look at the right subset of patients which the right type of disease, or develop a more targeted and more specific inhibitor, we will get even better results.
Which points us to where a lot of oncology is going now, which is toward "precision medicine". Some of the new cancer therapies are useful only if your tumor exhibits a specific genetic mutation. That is, it might not matter if you have lung cancer or breast cancer or colon cancer -- if your cancer has a mutation in XYZ gene, you should be on ABC inhibitor.[1] For some patients, the results for some of these therapies have been near miraculous. For some, it has not. But again, it gives us hope that there is a lot to be learned, as uncovering tumor DNA sequences has not been cheap or easy until relatively recently, especially since tumors exhibit heterogeneity in their DNA. Another hot area is personalized immunotherapy, as with CAR T-cell therapies, which harvest your body's immune cells and custom re-engineer them to attack your cancer -- definitely not a cheap prospect currently.
So as therapies get over the hurdles of showing "enough" benefit vs risk to get through the FDA, and as payers agree to reimburse for them, once again the research enterprise will increase its resources to these new paradigms. And with all that inertia behind it, sometimes the research field is over-excited about the uncovering of an previously unknown mechanistic link and the promise of new therapies... and we end up a little blind to the crappiness of the clinical studies and actual outcomes. I believe the system can be better. But as of now, the costs and the incentives and the burdens of heavily regulated R&D and the sick patients combine to make it a challenging landscape to work in, and alas that is currently how it works.
[1] One example is microsatellite instability / mismatch repair & a therapy called pembrolizumab, which was uncovered during a trial of patients with advanced colon cancer that was going very poorly, except for one patient who had an incredible response to the drug. Further investigation and subsequent studies demonstrated a link, reported in NEJM (http://www.nejm.org/doi/full/10.1056/NEJMoa1500596) and elsewhere, leading to eventual FDA approval for that indication.