I have heard that before, it's essentially the LLM cannot be creative packaged differently. But it always falls flat because there is just no way to measure it let alone define it. In the end it always give me a vibe of them wanting that to be the case rather than it being it case.
> But it always falls flat because there is just no way to measure it let alone define it.
I disagree. The definitions may have some gray areas, but at least looking back in history there are some breakthroughs that would fall firmly within the "a result found within the convex hull of ideas that existed at that time" type (e.g. Einstein's special theory of relativity), but also some truly novel, "out of the blue" breakthroughs that people agree are fully outside the convex hull of ideas from that time (e.g. Einstein's general theory of relativity).
Based on that, mathematicians and scientists should be able to provide some falsifiable hypotheses about the theorems and kinds of proof methods that would be truly novel.
I am not a research mathematician. But I think it's pretty clear you could get a bunch of research mathematicians in a room to more clearly define what a result looks like that is "outside of the convex hull", so to speak.
Most importantly, you don't need to define it 100%. You just need to come up with a bunch of specific examples where one would say "the proof of this would be outside the convex hull" and get wide consensus as to why, and then you'd have your falsifiable hypothesis.
Sure they can. Make the AI play the game pre and post bevy update. And difference down to a pixel can be flagged. In fact you can get hundreds of hours of testing this way using sub-agents.
Sure, the AI plays the game, says everything makes sense, does this mean it's coherent and consistent? No.
I'd urge you to try this yourself and you'll see how much they miss and misunderstand. They're nowhere near to being close to be able to do that sort of playtesting themselves.
Any game that been 100% made by LLMs in a vibe-coding way where the author does no testing themselves, will guaranteed be an absolutely mess and incoherent. This is the state of the art today at least, who knows what tomorrow will bring.
You’ve taken my statement, that an LLM can help test a game by running hundreds of hours of gameplay equivalence between versions(which btw means the AI doesn't say "it makes sense", your equivalence checker does), and stretched it into a claim that I’m saying games can be made entirely by LLMs through vibe coding with zero human testing. That’s such a blatant misrepresentation of my argument that I struggle to take your comment in good faith.
The difference would likely not be down to a pixel, either by dependence on how a model provider feels that day, the hallucinations of the day, or just some odd tangent that would get solved by ~100x tokens for verification. The hundreds of hours of testing could all effectively be 1===1, but it all looks green.
You have the model play both versions of the game in the same way at the same time. The equality would of course not come from the model, it comes from your testing harness. You just use the model to automatically play the game. SO you are checking game version A(inputs) == game version B(inputs). Where the AI generates the inputs that play through the game 10 times in different ways for example.
The testing harness would have to be updated per game feature and to accommodate model tendencies or quirks. This sounds like regression tests with llms and some extra steps.
Go is not higher level. If anything I would consider Rust higher on the abstraction ladder than Go. Or rather it's perhaps better to say that Rust is higher level but can go down to a lower level than Go.
I both agree with you and still think my initial idea was correct, and it means that I have expressed it very badly. What I meant by higher order was all the little niceties Go has that becomes a nightmare in Rust, the memory management, simple language; I mean just think about working with strings in Go vs Rust. The amount of boiler plate stuff always gives me a headache in Rust. But you're right, the map to C and C++ and go is orders of magnitude less developed in that terms.
So in a sense, cobol is a higher level language then both due to PIC Clauses? I don’t follow this if higher level languages is solely based on syntactic sugar.
I agree that "if higher level languages is solely based on syntactic sugar", then Rust is not any higher level than Go at least if you ignore the fact that Rust gives the coder the ability to define macros (which I'm happy to do because I don't like macros much).
The abstractions that Rust can express that Go cannot that I find very attractive are mostly restrictions: they give the coder a way to commit to behaving a certain way throughout the codebase. Sort of like OpenBSD's pledge and unveil system calls, but part of the language, not part of the OS. Of course the coder can back out of the commitment by removing the declarations that produced the commitment.
In general, any elaborate type system in any language (such as Rust has) can be fruitfully considered a way for a coder to impose restrictions on himself and any other coders editing the codebase for the benefit of anyone trying to understand the codebase.
Generally building something is not about "What does this teach us?" but rather "What can this be used for?". You don't build a house and as "What does this teach us?". It's honestly almost a perverse question to primarily ask for something that aims to have practical value.
The clear reason this has worth is that it's much faster than TSC.
HN is going to be a weird place to be between people loving AI and the loss of loving tech. I don’t remember last time I watched a good tech dev video. At least the video game industry is a little bit shielded from this but soon it will be the same when we see the speed AI is advancing.
I don't see at all that this is stifling love of tech. Sure if you liked to engage in language wars or vim vs emacs or tabs vs spaces then yes that's over. But tech as we know it is currently exploding, a ton of stuff that simply was not practically possible before is now possible.
Give us some examples of things that were not practically possible. Things are practically possible if the time or money is wisely spent, even if it takes a human a bit longer.
Money and time are precisely the point. Things are not practically possible of a X would cost 100K to develop and it's worth was 30K. But now that it costs 2K to develop it is suddenly practically possible.
2. The abundance of game decomps and recomps becoming available. Literally dozens of example in this space.
3. Reverse engineering of obscure software. A ton of oppertunities lie here.
4. As this posts indicate, severely accelerating existing software by re-writing it in language that are not terrible for performance.
5. Analyzing large volumes of unstructured information.
6. Creating one-off engineering and scientific tools.
7. Beating essentially all traditional Linux distribution, see Omarchy.
8. A personal example: I'm remaking the engine of an old game commercial game in rust.
I'm honestly a bit taken aback that this question is even asked. The volume of software is exploding and the power of the individual has increased a hundredfold.
I don’t quite follow. If you bring software worth into question, as you pointed out everyone’s vibe coding each of your examples so their individual worth has dropped significantly (with the unknown stability of support and long term plans).
My thought was LLMs accelerate the best and worst of development, but what couldn’t be done (outside of a large lab or company) by regular humans with an infinite time scale?
I.e. 8. A personal example: I'm remaking the engine of an old game commercial game in rust.
You don’t need an LLM for that, just some dedicated time for it.
Also your phrasing needs a lot of work. I recommend the Dale Carnegie books on communicating with others.
I think maybe 1 or 2 of those would take more than 1 year for an mvp. I will agree with you that with LLMs it’s infinitely easier and quicker to get a low quality mvp.
The point is that it's clearly not just a less maintained, less tested implementation. It's 1.61x faster for one in real world apps, in some case even almost 4x faster.
I am curious why it's faster. Golang isn't that much slower than Rust, aside from the garbage collector. 4x performance improvements is much bigger than what I expected.
> The comment I responded on put forth a generic argument that implied if we can not learn from it it has no value
Definitely not. They said they don't feel good using it because it's less maintained and less tested. They mentioned "we can't learn from it" to provide an example of a secondary benefit that this project could but (arguably) doesn't have. In other words, they had a pretty good reason not use this and didn't see much value if they couldn't use it.
You made it sound like they had some ultra extreme view that software is only valuable if you can learn from it.
I would expect it's the other way around. The AI wrote a Proof of NS in lean and write up a NL proof based on the lean one. The authors claim that openAI did NL -> Lean, but that is unsubstantiated.
> The agents arrived at their resolution on Saturday, September 5, about 88 hours after the first agents were launched. Lean formalization and verification took an additional 17 hours via GPT‑6 Astra.
This seems pretty clear that they found the proof first and then mechanized it afterwards.
The CPU is also in a bit of a funny place because the transpiled C++ looks to be translating instructions into manipulation of a MIPS CPU state object.
You always have to cost/quality of living if you go that route. Because that's literally what taxes are for. Anyway I don't think it's fair to not include VAT. For every euro you want to utilize(modulo some exceptions) you have to pay it.
reply