There are 3 new commits, and the only actually fixes are: Go update and revert to earlier version of console.
But there are a bunch of changes to docs, CI workflows and issue templates. Which is what is the easy part of managing a fork, and I've seen a bunch of forks that ended up only updating readme-s, CI, etc.
I'll have more faith in the fork when the maintainers do actual fixes.
Although, to be fair, getting too aggressive off the bat would be concerning. A clean fork that is bit for bit compatible with the last open source version is definitely an attractive proposition from a software supply chain perspective.
That's a good point. Though I'd still expect more effort from the developer to convince people that they will actively maintain the project after the first 3 months.
This does seem like a person getting hooked on idle games, or mobile/online games with artificially limited progress (that you can pay to lift). It's a type of delayed gratification that makes you anxious to get next one.
Not everyone gets hooked on those, but I do. I've played a bunch of those long-winded idle games, and it looks like a slight addiction. I would get impatient that it takes so long to progress, and it would add anxiety to e.g. run this during breaks at work, or just before going to sleep. "Just one more click".
And to be perfectly honest, it seems like the artificial limits of Anthropic (5 hour session limits) dig into similar mechanism. I do less non-programming hobbies since I've got myself a subscription.
It doesn't seem obvious that it's a problem for LLM coders to write their own tests (if we assume that their coding/testing abilities are up to snuff), given human coders do so routinely.
This thread is talking about vibe coding, not LLM-assisted human coding.
The defining feature of vibe coding is that the human prompter doesn't know or care what the actual code looks like. They don't even try to understand it.
You might instruct the LLM to add test cases, and even tell it what behavior to test. And it will very likely add something that passes, but you have to take the LLM's word that it properly tests what you want it to.
The issue I have with using LLM's is the test code review. Often the LLM will make a 30 or 40 line change to the application code. I can easily review and comprehend this. Then I have to look at the 400 lines of generated test code. While it may be easy to understand there's a lot of it. Go through this cycle several times a day and I'm not convinced I'm doing a good review of the test code do to mental fatigue, who knows what I may be missing in the tests six hours into the work day?
> This thread is talking about vibe coding, not LLM-assisted human coding.
I was writing about vibe-coding. It seems these guys are vibe-coding (https://factory.strongdm.ai/) and their LLM coders write the tests.
I've seen this in action, though to dubious results: the coding (sub)agent writes tests, runs them (they fail), writes the implementation, runs tests (repeat this step and last until tests pass), then says it's done. Next, the reviewer agent looks at everything and says "this is bad and stupid and won't work, fix all of these things", and the coding agent tries again with the reviewer's feedback in mind.
Models are getting good enough that this seems to "compound correctness", per the post I linked. It is reasonable to think this is going somewhere. The hard parts seem to be specification and creativity.
Maybe it’s just the people I’m around but assuming you write good tests is a big assumption. It’s very easy to just test what you know works. It’s the human version of context collapse, becoming myopic around just what you’re doing in the moment, so I’d expect LLMs to suffer from it as well.
> the human version of context collapse, becoming myopic around just what you’re doing in the moment
The setups I've seen use subagents to handle coding and review, separately from each other and from the "parent" agent which is tasked with implementing the thing. The parent agent just hands a task off to a coding agent whose only purpose is to do the task, the review agent reviews and goes back and forth with the coding agent until the review agent is satisfied. Coding agents don't seem likely to suffer from this particular failure mode.
One option is to do multi-stage rollout of your database schema and code, over some time windows. I recall a blog post here (I think) lately from some Big Company (tm) that would run one step from the below plan every week:
1. Create new fields in the DB.
2. Make the code fill in the old fields and the new fields.
3. Make the code read from new fields.
4. Stop the code from filling old fields.
5. Remove the old fields.
Personally, I wouldn't use it until I really need it. But a simpler form is good: do the required schema changes (additive) iteratively, 1 iteration earlier than code changes. Do the destructive changes 1 iteration after your code stops using parts of the schema. There's opposite handling of things like "make non-nullable field nullable" and "make nullable field non-nullable", but that's part of the price of smooth operations.
I had a strange problem with Dell P2720DC (27'' 2560x1440) - the whole outer edge of the screen would flicker if I used dark background with a lot of dim colours (like dark mode in IDE, or default Grafana theme). It wouldn't happen all the time, but it would happen on a weekly basis (I most often seen that on Sunday). I've RMA'd it, got another monitor, which also started showing the same problem. I gave it to someone who doesn't use dark mode - no issues.
So, I'm not getting another Dell until I'd be sure this issue won't happen again :)
NX NPM attack (at least the previous wave which targetted tinycolor) relied on running post-install scripts. Go tooling does not give you ways to run post-install scripts, which is much more reasonable approach.
Github Actions Runner code is pretty easy to read, here's a specific place that define default arguments for popular shells / binaries: https://github.com/actions/runner/blob/main/src/Runner.Worke..., it is exported through a method ScriptHandlerHelpers.GetScriptArgumentsFormat.
In ScriptHandler.cs there's all the code for preparing process environment, arguments, etc. but specifically here's actual code to start the process:
Overall I was positively surprised at simplicity of this code. It's very procedural, it handles a ton of edge cases, but it seems to be easy to understand and debug.
LLMs (our current "AI") doesn't use logical or mathematical rules to reason, so I don't see how Gödel's theorem would have any meaning there. They are not a rule-based program that would have to abide by non-computability - they are non-exact statistical machines. Penrose even mentions that he hasn't studied them, and doesn't exactly know how they work, so I don't think there's much substance here.
Well, if you break everything down to the lowest level of how the brain works, then so do humans. But I think there's a relevant higher level of abstraction in which it isn't -- it's probabilistic and as much intuition as anything else.
Nothing to do with it? You certainly don’t mean that. The software running an LLM is causally involved.
Perhaps you can explain your point in a different way?
Related: would you claim that the physics of neurons has nothing to do with human intelligence? Certainly not.
You might be hinting at something else: perhaps different levels of explanation and/or prediction. These topics are covered extensively by many thinkers.
Such levels of explanation are constructs used by agents to make sense of phenomena. These explanations are not causal; they are interpretative.
> Nothing to do with it? You certainly don’t mean that. The software running an LLM is causally involved.
Not in the way that would apply problem of non-computability of Turing machine.
> Perhaps you can explain your point in a different way?
LLM is not a logic program finding perfect solution to a problem, it's a statistical model to find next possible word. The model code does not solve a (let's say) NP problem to find solution to a puzzle, the only thing is doing is finding next best possible word through statistical models built on top of neural networks.
This is why I think Gödel's theorem doesn't apply here, as the LLM does not encode strict and correct logical or mathematical theorem, that would be incomplete.
> Related: would you claim that the physics of neurons has nothing to do with human intelligence? Certainly not.
I agree with you, though I had different angle in mind.
> You might be hinting at something else: perhaps different levels of explanation and/or prediction. These topics are covered extensively by many thinkers.
> Such levels of explanation are constructs used by agents to make sense of phenomena. These explanations are not causal; they are interpretative.
A lot of people look towards non-determanism to be a source for free will. It's often what underlies peoples thinking when they discount the ability of AI to be conscious. They want to believe they have free will and consider determinism to be incompatible with free will.
Events are either caused, or uncaused. Either can be causes. Caused events happen because of the cause. Uncaused events are by definition random. If you can detect any real pattern in an event you can infer that it was caused by something.
Relying on decision making by randomness over reasons does not seem to be a good basis of free will.
If we have free will it will be in spite of non-determinism, not because of it.
That's true with any neural network or ML model. Pick a few points, use the same algorithm with the same hyperparameters and random seed, and you'll end up with the same result. Determinism doesn't mean that the "logic" or "reason" is an effect of the algorithm doing the computations.
Not really possible. The models work fine once you fix them, it's just making sure you account for batching and concurrency's effect on how floating point gives very (very) slightly different answers based on ordering and grouping and etc.
Good point, I meant the reasoning is not encoded like a logical or mathematical rules. All the neural networks and related parts rely on e.g. matrix multiplication which works by mathematical rules, but the models won't answer your questions based on pre-recorded logical statements, like "apple is red".
If it is running on a computer/Turing machine, then it is effectively a rule-based program. There might be multiple steps and layers of abstraction until you get to the rules/axioms, but they exist. The fact they are a statistical machine, intuitively proves this, because - statistical, it needs to apply the rules of statistics, and machine - it needs to apply the rules of a computing machine.
The program - yes, it is a rule-based program. But the reasoning and logic responses are not implemented explicitly as code, they are supported by the network and encoded in the weights of the model.
That's why I see it as not bounded by computability: LLM is not a logic program finding perfect solution to a problem, it's a statistical model to find next possible word.
But there are a bunch of changes to docs, CI workflows and issue templates. Which is what is the easy part of managing a fork, and I've seen a bunch of forks that ended up only updating readme-s, CI, etc.
I'll have more faith in the fork when the maintainers do actual fixes.
reply