That's a bizarre takeaway for them to suggest, when they had exactly the same kind of bug with Rust like three weeks ago. (In both cases they had code implicitly expecting results to be available. When the results weren't available, they terminated processing of the request with an exception-like mechanism. And then they had the upstream services fail closed, despite the failing requests being to optional sidecars rather than on the critical query path.)
In fairness, the previous bug (with the Rust unwrap) should never have happened: someone explicitly called the panicking function, the review didn't catch it and the CI didn't catch it.
It required a significant organizational failure to happen. These happen but they ought to be rarer than your average bug (unless your organization is fundamentally malfunctioning, that is)
It's different to expect somebody to write the correct program every time than to expect somebody not to call the "break_my_system" procedure that was warnings all over it telling people it's there for quick learning-to-use examples or other things you'll never run.
To be precise, the previous problem with Rust was because somebody copped out and used a temporary escape hatch function that absolutely has no place in production code.
It was mostly an amateur mistake. Not Rust's fault. Rust could never gain adoption if it didn't have a few escape hatches.
"Damned if they do, damned if they don't" kind of situation.
There are even lints for the usage of the `unwrap` and `expect` functions.
As the other sibling comment points out, the previous Cloudflare problem was an acute and extensive organizational failure.
You can make an argument that .unwrap() should have no place in production code, but .expect("invariant violated: etc. etc.") very much has its place. When the system is in an unpredicted and not-designed-for state it is supposed to shut down promptly, because this makes it easier to troubleshoot the root cause failure whereas not doing so may have even worse consequences.
The correct solutions and the viable paths probably are known to the trainers, just not to the trainee. Training only on problems where the solution is unknown but verifiable sounds like the ultimate hard mode, and pretty hard to justify unless you have a model that's already saturated the space of problems with known solutions.
(Actually, "pretty hard to justify" might be understating it. How can we confidently extract any signal from a failure to solve a problem if we don't even know if the problem is solvable?)
Your hard mode is exactly the situation that RL is used, because it requires neither a corpus of correct examples, nor insight into the structure of a good policy.
> How can we confidently extract any signal from a failure to solve a problem if we don't even know if the problem is solvable?)
You rule out all the stuff that doesn’t work.
Yes this is difficult and usually very costly. Credit assignment is a deep problem. But if you didn’t find yourself in a hard mode situation, you wouldn’t be using RL.
That is a bizarre take. Dwarkesh Patel is publishing in a very specific domain, where RL is a very common and unambigous acronym. I'd bet it was immediately clear to 99% of his normal audience, and to him it's such a high frequency term that people finding it ambiguous would not even have crossed his mind.
(Like, would you expect people to expand LLM or AGI in a title?)
That's not the logic, they obviously hire from outside. The author's complaint is not that he can't get hired. He doesn't want to get hired, even! The complaint is rather that investors aren't funding his startup.
There is nowhere near enough money in the ad network business. Like, Google's search ad business is an order of magnitude higher than the ad network, and the ad network has been shrinking in absolute terms for years while the first party revenue has been growing at double digits.
Spending a lot (on capex or opex) certainly is not providing any kind of signaling benefit at this level. It's the opposite, because obviously every single financial analyst in the market is worried about the rapid increase in capex. The companies involved are cutting everything else to the bone to make sure they can still make those (necessary) investments without degrading their top-line numbers too much. Or in some cases actively working to hide the debt they're financing this with from their books.
Even if we imagined that the author's conspiracy theory were true, there would still be massive incentives for optimization because everyone is bottlenecked on compute despite expanding it as fast as is physically possible. Like, are we supposed to believe that nobody would run larger training runs if the compute was there? That they're intentionally choosing to be inefficient, and as a result having to rate-limit their paying customers? Of course not.
The reality is that any serious ML operation will have teams trying to make it more efficient, at all levels. If the author's services are not wanted, there are a few more obvious options than the outright moronic theory of intentional inefficiency. In this case most likely that their product is an on-edge speech to text model, which is not at all relevant to what is driving the capex.
> Spending a lot (on capex or opex) certainly is not providing any kind of signaling benefit at this level.
It's not providing any benefit now but there's still signalling going on, and it absolutely provided benefit at the beginning of this cycle of economy-shattering fuckwittery.
ChatGPT usage is already significantly higher than Twitter at its peak, and there is a lot more scope activity with explicitly or implicitly commercial intent. Twitter was entertainment and self-promotion. Chatbots are going to be asked for advice on how to repair a dish washer, whether a rash is something to worry about, which European city with cheap flights has the best weather in March for a wedding, and an indefinite stream of other similar queries.
I don't share your optimism. Those kinds measures would be just security theater, not "a lot better".
Avoiding secrets appearing directly in the LLM's context or outputs is trivial, and once you have the workaround implemented it will work reliably. The same for trying to statically detect shell tool invocations that could read+obfuscate a secret. The only thing that would work is some kind of syscall interception, but at that point you're just reinventing the sandbox (but worse).
Your "visually inspect the contents of the URL" idea seems unlikely to help either. Then the attacker just makes one innocous-looking request to get allowlisted first.
reply