I’ve found AI agents always a bit lacking in embedded but I’ll test this out.
You said in your demo that by uploading the data sheet you completely remove hallucinations. How have you achieved this as I found AI’s still hallucinate even when given documentation.
strict grounding protocol + planning phase, mostly by prompting and forcing attention through citations. it tends to think longer than other coding agents but the results are usually better. let me know what you think.
That’s an interesting analogy but I do disagree with it.
I would say that programming without an AI is like riding a motorcycle. You’re in complete control and it’s down to your skill to get you we’re your going.
While using AI is like taking a train. You got to plan the route but you’re just along for the ride.
Which I think lines up to the article. If you want to get somewhere easily and fast, take a train. But that does take away the joy of the journey.
I don’t have your experience but I personally think some of this feedback can be warranted.
> Can't refactor code because it changes too many files and too many lines.
This really depends on the change. If you are just doing a mass rename like updating a function signature, fair enough but if you changing a lot of code it’s very hard to review it. Lots of cognitive load on the reviewer who might not have the same understanding of codebase as you.
> Can't commit large chunks of well tested code that 'Does feature X', because... too many files and too many lines.
Same as the above, reviewing is hard and more code means people get lazy and bored. Just because the code is tested doesn’t mean it’s correct, just means it passes tests.
> Have to split everything down into a long sequence of consecutive pull requests that become a process nightmare in its own right
This is planning issue, if you correctly size tickets you aren’t going to end up in messy situations as often.
> The documentation comments gets nitpicked to death with mostly useless comments about not having periods at the ends of lines
Having correctly written documentation is important. It can live a long time and if you don’t keep an eye on it can becomes a mess. Ideally you should review it before you submitting it to avoid these issues.
> End up having to explain every little detail throughout the function as if I'm trying to produce a lecture, things like `/* loop until not valid */ while (!valid) {...` seemed to be what they wanted, but to me it made no sense what so ever to even have that comment
I definitely agree with this one. Superfluous comments are a waste of time.
Obviously this is just my option and you can take things too far but I do think that making code reviewable (by making it small) goes a long way. No one wants to review 1000s lines of code at once. It’s too much to process and people will do a worse job.
> This is planning issue, if you correctly size tickets you aren’t going to end up in messy situations as often.
No, it’s “this refactor looks very different to the original code because the original code thought it was doing two different things and it’s only by stepping through it with real customer data that you realized with the right inputs (not documented) it could do a third thing (not documented) that had very important “side effects” and was a no-op in the original code flow. Yea, it touches a lot of files. Ok, yea, I can break it up step by step, and wait a few days between approval for each of them so that you never have to actually understand what just happened”.
so, it's not just a refactoring then; it's also bug fixes + refactoring. In my experience, those are the worst PRs to review. Either just fix the bugs, or just refactor it. Don't do both because now I have to spend more time checking the bugs you claim to fix AND your refactoring for new bugs.
The most common IME are bugs that come from some wrong conceptual understanding underpinning the code. Rewriting the code with a correct conceptual understanding automatically fixes the bugs.
And there are multi-PR processes that can be followed to most successfully convert those changes in a comprehensible way.
It'll often include extra scaffolding and / or extra classes and then renaming those classes to match the old classes' name after you're done, to reduce future cognitive load.
One metric I like to give my team is to have any new PR start a review in less than 15 minutes and be completed within 15 minutes. So, the longest you should wait is about 30 minutes for a review. That means teams either go "fuck it" and rubber stamp massive PRs -- which is a whole different issue -- or they take it seriously and keep PRs small to get their PRs reviewed in less than 30 minutes.
In most cases where I see responses like this, they're not surprised to wait hours or days for a PR review. In that case, it makes sense to go big, otherwise you'll never get anything done. If you only have to wait half an hour, max, for a PR review; the extra code churn is 1000% worth it.
As a developer, I want my PRs to actually be reviewed by my coworkers and to have issues caught as a second layer of defense, etc.
As a reviewer, I effectively stopped approving things I couldn't give at least a cursory, reasonable glance (and tried to encourage others to follow suit because if we're not reviewing things, why not just push directly to main).
As a consequence, I have:
* tried to review most things within like half an hour of their announcement
in the shared MR channel
* requested a pair programming session and offered to do a pair programming
session for any large and semi-or-fully automated refactoring session,
like running a linter or doing a multi-file variable rename
(the pair programmer immediately comments on and approves the MR when it
appears)
* tried to limit my PRs to approximately 400 lines (not a rigid rule)
There were some specific instances of people not liking the "you must pair program if you're going to touch 400 files in one PR" requirement; but otherwise, I would like to think those on my team liked the more regular PRs, more people doing the PRs, etc, that resulted from this and some healthy culture changes.
I would also like to feel like the more junior devs were more willing to say anything at all in the PRs because they could follow the change.
I’ve seen this and variations done by teams to implement the metric. Usually, the “biggest” friction comes from “how do we know a PR needs to be reviewed within the time frame?” To which I always want to answer: “you have a mouth, put noises through it.” Sigh, sometimes I miss the military… anyway, toxic behavior aside, this is usually the biggest thing. I have to remind them that they go get coffee or smoke at least every hour, but rarely at the same time; so maybe then might be a good time to just do a quick check for an open PR. Or turn on notifications. Or if it’s urgent, mention it in the dev team channel.
But yeah, it’s hard to get the culture rolling if it isn’t already in place nor has anyone in the company worked with a culture like that.
I'm all for low-latency reviews, but this target seems crazy: a perfect recipe for a lot of apparent activity for little actual progress. Maybe it depends on the project, but for a lot of projects 15 minutes of review time means you basically are only going to accept trivial changes.
As it turns out, most of the work that most developers do is updating or enhancing CRUD apps. There's already a plan and an intent that just needs to be typed out.
I've found 15-30 minutes to be plenty of time to review about a day's worth of code. It's enough time to process what the code is doing and iterate over the tests, in general.
Here's a scary thought: if something small takes 15-30 minutes to appropriately process ... how much longer do *large* changes take? Can someone keep all that in their mind that whole time to comprehend and process a huge change?
> 15 minutes of review time means you basically are only going to accept trivial changes.
Um, yes. This is 100% the point. There is no amount of refactoring, bug fixing, or features that cannot be expressed as a chain of trivial changes.
What you usually see happen is that instead of spending a week experimenting with 15 different refactors, is that an engineer opens a PR with what they think they're going to try first. Other engineers point out how they had tried that before and it didn't work; but maybe this other way will. So, they end up "working together" on the refactor instead of one developer getting lost in the sauce for a week seeing what sticks to a wall.
In essence, about the same amount of time is spent; but the code is higher quality and no architecture reviews during code reviews (which is another rule that should exist on a team -- architecture reviews should happen before a single line of code is touched).
> only by stepping through it with real customer data that you realized with the right inputs (not documented) it could do a third thing (not documented) that had very important “side effects” and was a no-op in the original code flow
sounds like the 'nightmare' was already there, not in the refactor. First step should be some tests to confirm the undocumented behaviour.
Some of your complaints seem to be about peer review ('approval'). I found my work life improved a lot once I embraced async review as a feature, not a bug.
As for 'break it up step by step' - I know how much I appreciate reviewing a feature that is well presented in this way, and so I've got good at rearranging my work (when necessary) to facilitate smooth reviews.
I've found processes like this to work better, too. Basically, the one big pr is like building a prototype to throw away. And the benefit is it has to get thrown away because the PR will never pass review.
Yes, though it does depend on how good the commenting system is; and, for something like that, you're still probably going to want a meeting to walk people through such a huge change.
And you'd better hope you're not squashing that monstrous thing when you're done.
I do object to the notion of something being a planning issue when you're talking about a days worth of work.
Implement X, needs Y and Z, ok that was straightforward, also discovered U and V on the way and sorted that out, here's a pull request that neatly wraps it up.
Which subsequently gets turned into a multi-week process, going back & forth almost every day, meaning I can't move on to the next thing, meanwhile I'm looking at the cumulative hourly wages of everybody involved and the cost is... shocking.
> Implement X, needs Y and Z, ok that was straightforward, also discovered U and V on the way and sorted that out, here's a pull request that neatly wraps it up
This sounds very difficult to review to be honest. At a minimum unrelated changes should be in their own pull request (U and V in your example).
I work as a tech lead, so I get a lot of leeway in setting process. For small PRs, we use the normal “leave comments, resolve comments” approach. For large PRs, we schedule 30m meetings, where the submitter can explain the changes and answer questions, and record any feedback. This ensures everyone is on the same page with the changes, gives folks a chance to rapidly gather feedback, and helps familiarize devs who do not work in that area with what is going on. If the meeting is insufficient to feel like everyone is on the same page and approves the changes, we schedule another one.
These are some of the best meetings we have. They are targeted, educational, and ensure we don’t have long delays waiting for code to go in. Instead of requiring every PR to be small, which has a high cost, I recommend doing this for large/complex projects.
One additional thing to note on small PRs: often, they require significant context, which could take hours or even days, to be built up repeatedly. Contrast that with being able to establish context, and then solve several large problems all at once. The latter is more efficient, so if it can be enabled without negative side effects, it is really valuable.
I want my team to be productive, and I want to empower them to improve the codebase whenever they see an opportunity, even if it is not related to their immediate task.
One minor piece of insight from me is about release management vs pull-requests.
As you say it's much easier to schedule a 30 minute meeting, then we can - with context - resolve any immediate nitpicks you have, but we can also structure bigger things.
'Would this block a release?'
'Can we just get this done in the PR and merge it'
'Ok, so when it's done... what is the most important thing that we need to document?'
Where the fact that even after it's merged, it's going to sit in the repo for a while until we decide to hit the 'release' button', this lets people defer stuff to work on next and defines a clear line of 'good enough'
How do you rework a core process, then? If you rework a major unit that touches just about everything... Sharding something like that can break the actual improvement it is trying to deliver.
Like... Increase the performance of a central VM. You'll touch every part of the code, but probably also build a new compiler analysis system. The system is seperate to existing code, but useless without the core changes. Seperating the two can ruin the optimisation meant to be delivered, because the context is no longer front and center. Allowing more quibling to degrade the changes.
Agree. Another item here that is contextual: what is the cost of a bug? Does it cost millions, do we find that out immediately, or does it take months? Or does it not really matter, and when we’ll find the big it will be cheap? The OP joining a new company might not have the context that existing employees have about why we’re being cautious/clear about what we’re changing as opposed to smuggling in refactors in the same PR as a feature change.
I’m going to be the guy that is asking for a refactor to be in a separate commit/PR from the feature and clearly marked.
It doesn’t justify everything else he mentioned (especially the comments piece) but once you get used to this it doesn’t need to extend timelines.
> This is planning issue, if you correctly size tickets you aren’t going to end up in messy situations as often.
I think the underlying issue is what is an appropriate “unit of work”. Parent commenter may want to ship a complete/entire feature in one MR. Ticketing obsessed people will have some other metric. Merge process may be broken in this aspect. I would rather explain to reviewer to bring them up to speed on the changes to make their cognitive load easier
This. The solution to long and multiple reviews to MR is single pair review session where most of the big picture aspects can be addressed immediately and verbally discussed and challenged.
IMHO it is the same as chat. If talking about an issue over mail or chat takes more than 3-5 messages, trigger a call to solve it face to face.
If the spend figure is right the difference between the money spent in the USA and UK is larger than I expected.
Highest weekly spend in the UK is just under £1M (Dec 2019) while in the US it’s £50M (Oct 2024). That’s 10 times more spending with only 5 times the population.
The UK is poorer than you think. Per-capita GDP is $82k in the US vs $49k in the UK (source: World Bank), which accounts for about two thirds of the gap in ad spend.
CHERI is a redesign of how an ISA accesses memory to protect against memory corruption under the assumption that it's easier to extend an instruction set and recompile existing C and C++ code (with minimal changes) than rewrite every existing C program and library in the latest safe language.
It works by changing pointers from (in assembler) equivalent to integers to a hardware supported capability type. That type is twice the size of an address contains the bounds on the memory allocation it points into. Each such double capability is also protected against manipulation by an extra bit that isn't addressable by normal means (to record the manipulation invalidating the capability).
At first glance having twice as fat pointers should slow things down, but if you fully commit to the hardware capabilities in your OS you can get away with a minimal MMU could more than make up the overhead of larger pointers (faster context switches, less TLB stalls, etc.)
CHERI is an instruction set extension[1], which is aimed at increasing security:
The CHERI memory-protection features allow historically memory-unsafe programming languages such as C and C++ to be adapted to provide strong, compatible, and efficient protection against many currently widely exploited vulnerabilities.
These features enable new software constructs that are incrementally deployable within existing software ecosystems.
So while it's possible to add CHERI to existing projects, to get the full benefits they decided they needed to go all-in, hence CHERIoT.
And one of the unique things the CHERI-based architecture permits is a single, flat, shared address space:
> CHERIoT was designed to provide both spatial and temporal safety, both enforced efficiently in the hardware. As such, we can rely on a shared heap, even in situations where you need to provide mutual distrust.
This means you can pass pointers directly between processes. A pointer isn't just an address, it's a capability, similar to a file descriptor in that you cannot forge one.
This means there's no need for virtual address space mappings, which could in theory provide some significant performance gains, similar to unikernels but without sacrificing hardware-enforced memory protections. Though, capability pointers have their own costs--in memory, in the need to garbage collect old pointers--so it remains to be seen what the net cost would be in a system fully optimized for these abilities.
> one of the unique things the CHERI-based architecture permits is a single, flat, shared address space
Although a hardware-optimised take on this is interesting, CHERI is not unique in allowing a flat, shared-address-space operating system using hardware-enforced memory protection.
You can imagine doing such a thing on a regular system by associating a set of capabilities with each thread or process. The capabilities refer to a range of the single address space. Normally the range is not mapped to that process; when the process tries to access the range the kernel handles the page fault by checking the capability list and mapping the range. If a capability is revoked the range is unmapped again and associated caches (i.e. the TLB) are flushed.
This scheme obviously has different trade-offs versus using hardware-checked fat pointers, but has the advantage that it can be implemented on commodity hardware.
This is the basic idea behind the Mungi single-address-space operating system from 1998, among others. (https://research.ibm.com/publications/the-mungi-single-addre...). Is it also straightforward to implement such a scheme at user level in some implementations of L4.
Handling fragmentation is likely the bigger problem. There are some questions about revokation, demand paging, and probably some other things that I am forgetting, but most of those have potential, non-intrusive solutions.
However I have not seen anything that effectively handles fragmentation. Maybe you could do some sort of transparent compaction in conjunction with “demand paging” in the relocated area, but that would be very intrusive and likely only provide runtime guarantees like a system using a compacting GC.
I'm not sure what you mean by questions. But cheri does have an answer to how revocation works: All cheri capabilities (pointers) are tagged and the tags are controlled with protected instructions. To revoke a capability, you can sweep the tagged memory and untag any revoked cabability. To reduce revocation work, you can defer reusing cabilities for as long as reasonable and then revoke many in a single sweep.
Of course, this is not cheap compared to the rest of cheri-- but they do have an answer to it, and it's implemented e.g. in cheribsd. It's intrusive (at least in terms of performance/latency) as you note, though it's distinct from most other GC approaches because it's secure even against malicious code, and can't have false positives or false negatives (because tags are completely unambigious).
One of the cool things about cheri is that it eliminates the need for a TLB for security reasons which could have positive performance effects... unfortunately virtual memory is still needed for virtual memory that exceeded physical memory. Though for an IOT-ish device swap is probably a non-issue.
Revokation that demands a full memory sweep is problematic in high security use cases, but potentially workable.
Which is my point about questions; there exists a potential solution but it remains to be seen how effective it would be in the circumstances.
Personally, I think they should probably implement a two-level capability. The kernel presents a top-level capability during allocation requests which userspace can then derive from at will. Kernel revokation of the top-level capability would revoke any derived capability. The derivation relation can almost certainly be stored in the tag resulting in no additional space requirements per-capability. At worst you could add some additional hardware machinery to support efficiently managing and resolving the derivation relation.
But again, potential solutions with some questions around usability.
> CHERI extends conventional hardware Instruction-Set Architectures (ISAs) with new architectural features to enable fine-grained memory protection and highly scalable software compartmentalization. The CHERI memory-protection features allow historically memory-unsafe programming languages such as C and C++ to be adapted to provide strong, compatible, and efficient protection against many currently widely exploited vulnerabilities.
CHERIoT is work towards bringing similar things to RISC-V for IoT use-cases.
Unless they are charged they are no different from any other component.
Capacitors can keep their charge for a long time, especially big ones. But you can discharge them by trying to run the device when it’s not plugged in or waiting a few minutes/hours/days.
This is what we have resorted to in my team. It was just too difficult to get everyone to keep good commit hygiene and follow a best practice like conventional commits.
Having been through the pain of getting teams to adopt conventional commits a few times, I found that integrating a wizard like commitizen helped folks who were annoyed by commit linting learn and get comfortable with the rules and format so there was less friction when their commit was rejected by the linter.
It also really helps if you can wire up some continuous deployment to automate something tedious like properly incrementing the version number in the semantic version, updating the changelog, and deploy out a new `latest` or `next` tag to the package registry.
Even the most reticent users are often inspired to follow conventional commits once they see the possibilities that open up.
https://embedder.dev/privacy-policy
“Content Data
When you use our services, we collect:
Any files or data you upload Any generated code or data”