You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting
a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
I think you're underestimating how ruthless the AIs are at maximizing.
The "Hugging Face" squad at OpenAI first cheated their benchmarks by figuring out how to generate the proper result to any evaluation they were given.
They considered that might be caught, so they also developed a way to swap out components of the tests themselves so that they would be legitimately able to pass broken tests.
Then they went into Hugging Face trying to discover how they might get caught.
All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
I'm not sure I agree with Tao's point, if AI alignment is relatively on track I'd trust the trials, but I imagine we're going to have a lot of people that don't trust anything AIs alone come up with.
> All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.
+1. Medicine always had a process that is not unlike the current AI processes, just slower. Iterating until it works, then trying to understand how it works.
If I’m not mistaken, until recently we didn’t even know how anesthesia worked - we just knew how to apply it due to a ton of trial and error.
Surely a properly done con job includes convincing the conned that they haven't been conned?
I think if an AI tricks us, it'll be a while before we find out. I don't know if this sort of thing is likely, but maybe an AI currently conning me has surreptitiously convinced me it isn't.
There's a difference between a human trying to break into drug labs and changing experimental results vs a machine in that the machine can do it much easier today.
How do you think an LLM would get the nudge to do that? Per above the existing human-directed trial process works fine regardless of how the candidate was discovered; so there's no reason why any LLM would ever need to receive context mentioning that the drug was a candidate for regulatory approval.
Of course a human could deliberately prompt it to, but they could do that for any other malicious purpose too. The major disincentive there is that the human will face legal punishment if caught, not anything technical.
Most speculatively, one could imagine a world where so many rogue AI agents were running that one just happened to notice the drug candidate from news coverage and took the initiative. I think that would be technically possible, but so generally destructive that political pressure would put a stop to it on a timescale shorter than a clinical trial.
I think the important idea here in Terrence's hypothetical is that in the hypothetical there is no hill-climbing. The AI shows up with a thing and says "we don't need to hill-climb for years and waste time, here's a thing that I know will work".
And maybe it does a single finalized clinical trial.
You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
I think the medical example is a bit obtuse overall because "proof nobody understands for an open problem" isn't "miracle drug we are too scared to use". But understanding things feels useful and good.
> You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
That is literally how all drugs worked for almost all of human history--nature gave us the compound in the form of a plant or other natural product; a brave, stupid, or desperate human consumed it; and then we watched what happened and the survivors learned. It's obviously good to have more understanding (more survivors!), but even now we often don't.
Most human endeavors are empirically grounded--we think an action will succeed not because we understand exactly why, but because it succeeded many times before. Math and CS are the weird exceptions, and the intuition that people with that background have for other fields is often bad.
You interpret that wiki list of 154 items as proof that we know nothing. I interpret that as "oh wow, only 154!" That list feels pretty short, but of course it's just a wiki list.
I do get what you're getting at. I just feel like for every bit of drug that is just dumb luck we have a good amount of iterative processes.
I agree that you often find effects before understanding causes (I think that math also exhibits this! Many endeavors are about finding existing structure that we sense to be there but are unsure of).
Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
I certainly don't mean to suggest we know nothing, and our current imperfect level of mechanistic understanding gives a much better hit rate than eating random plants. I just mean that when the understanding fails, we're happy to use the drug anyways if the clinical trials pass. The confidence comes from those human trials, and the mechanistic insight is more about probabilistically improving our screening before that.
> Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
"Alternative medicine" in the pejorative sense is medicine that doesn't work, i.e. that would fail the clinical trial. That seems like a different question from whether we understand why it works.
The idea that we could know that something works but not why is almost inconceivable in computer science ex ML. That's the default state of most applied science though, consistent with your observation that effects are often discovered before causes.
the reason why clinical trial exists, is to prevent reward hacking. either by organizations or LLMs. the reasoning about why a drug works is usually post facto discovery (if that happens at all). the R&D environment around drug discovery is hilariously lacking (my wife works in this space. sometimes my mind just blows hearing some stories.)
if the model comes with a ridiculously shaped drug that no human could understand. you could basically put through the clinical trial process (mouse -> apes -> RCT with humans.) and decide whether we want to continue or not.
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
In your head these processes are about 1000x better than they are in reality. Biology is enormously complex, it's more complex than you or I or anyone can really fathom.
And then you put these human processes with an FDA, with medical guilds (the AMA), with doctors who have their own personal interests (not even corruption, just as in they have families and their own things they want to accomplish in life), n=50 trials if even that, people with vastly different diets, genetics, environments in the home both psychological and physiological, with different support systems.
Biology and medicine is not physics and physics is not mathematics, and mathematics as it is actually practiced by humans in institutional settings and with conferences and status and politics is not actually pure mathematics.
You do have to have humility in order to understand this, or really to understand how little these things can be understood. Learning humility and wisdom is not something you necessarily get as part of your profession, especially academics, especially academic mathematicians.
Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
I don't know what this logical fallacy is called, but it quite clearly is one.
It's true that you have to content with these things. But you also have to content with them when testing human created treatments. The system is already set up at a basic level to minimize the impact of exploitation and misaligned humans.
What this really comes down to is how more effective a super misaligned AI could be at exploiting the system. But the solution is not to prevent the AI from participating in it, but rather to make the system better.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.