so, i've gone through a bunch of iterations on how to interview in days of LLMs and i came up with a process that is quite good:
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
I feel like git operations are one of the few things that I truly don't care at all about whether someone uses an LLM for and doesn't tell me. PR descriptions and commit messages should be written by humans for humans, but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care? Beyond the code itself, I care about the things that will continue existing in the long run fitting my team's conventions (the number of commits, the commit messages, the PR description and comments, and then whether there's a merge commit at the end), but the process used to create those things doesn't matter to me.
Wasn't this a think before_ai? Enough people are using IDEs which handle this for them. I still have to check a few commands now and then.
Though ideally interviewers should appreciate this - unless its a hard requirement you do it by hand day to day - the rest of us can look this up quick enough.
At previous jobs, I was the one coworkers would come to when they had messy git problems and needed help cleaning them up. I've put in the hours in the past to learn how to do all of that, and then later forgotten it because it's just not worth it anymore. I switched to jj a few years ago and had the majority of the annoying rebase issues go away, and whatever's left goes into my LLM to spit out a potential solution that I can then manually tweak if the result isn't perfect.
My OS professor in college once told us about what he considered the difference between "knowledge" and "information", with "knowledge" being the underlying facts about how something works and "information" being the terminology/labeling/bucketing/etc. of how to refer to it. The point he made was that knowledge was more important than information, and that he would try to test us on knowledge rather than information. To me, knowing how the underlying version control model works and what "shapes" you might want to make with it is useful knowledge. The exact set of commands, interactions with an IDE, or specific prompts you can use to ask an LLM to produce them is information. Arguably the entire benefit of LLMs is that you can ask them to do things even if you don't have the exact information memorized; if you can describe it in a rigorous way, and recognize whether what it gives you as an output fits that or not, you don't need to know any of the actual information of how to do it manually.
> but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care?
Because LLMs aren't a reason to not learn your tools. A determinist CLI command vs using a statistical machine for it. It's just a waste of tokens at the very least.
No, the reason not to learn your tools is if they're not your tools anymore, and something else is. I don't spend time trying to manually invoke filesystem APIs to create files in my work projects and instead will often just have my LSP do it when I declare a module and the file isn't found. I could memorize the exact sequence of system calls needed, but my employer would prefer I spend time on things that are more worthwhile to them.
I don't disagree that having a deterministic CLI is useful, but I'd argue that's not the question here, because the CLI is what the statistical machine will be using. The question is whether the time spend doing it manually is worthwhile compared to the tokens spend on it. I'd argue that if you instead learn how to pick the right model for the right job so that you don't use something way more expensive than needed for the task at hand, the token overhead will be extremely minimal for stuff like git operations; knowing that you can use something cheaper than like, Astra or Opus if all you want to do is a quick git cleanup and push is going to save your employer a lot more money in the long run than if you insist on doing every messy rebase by hand.
While using LLMs is a great idea, it has become crucial to put safeguards in place to ensure that their use doesn't deviate from team practices or compromise security.
I'd argue that these are not really specific to git (or even LLMs honestly); in terms of security, don't put company data into anything that your company hasn't given you explicit permission to put it into.
In terms of team practices, that basically seems like a code review thing. If my team has conventions around git, and a PR doesn't fit them, that's something I would point out regardless of how the PR was generated. If a team member is repeatedly not meeting the expectations we've all agreed upon, I would deal with it in the same way I always have; reach out and offer guidance for how to do things the way they're supposed to, notify my manager if they continue having trouble with it, and eventually just not bother looking at their PRs if they absolutely refuse to do things the way everyone else on the team has agreed they should be done.
In the before time, this is the sort of thing people would just google.
One time setup commands are the sort of thing people do once when setting up the repo, but dont use day to day. its understandable that people dont have that memorized and will reach for a man page, google or an AI.
I happen to have memorised how to add new remotes a long time ago, but I would 100% not care if someone I worked with googled it every time. That's how it was for me for several years. I don't think that has anything to do with "learning the tools". And getting claude to do it is not that much different from copy-pasting from google.
Some 6-figure per year developer spending a quarter’s worth of tokens to do something they do once a month or so is not something I’ll ever care to optimize away…
That's pretty much where I'm at, so long as they're not filling the code with "///CLAUDE ADDITION" type guards (ie noise).
Personally I ask the LLM to summarise what the commit does and explain it to me, make sure I understand it, then review the PR. I'm getting half the speed the High Speed Sloperators are getting but I was the only person submitting PRs the other day when the LLM service was down.
For nasty multi-way git merges and rebases, it's been a game changer. And of course that's reviewable by doing a delta on the final rebase branch vs. the original branch.
With all that said... having to use LLMs at work is making me want to use them less on my own projects.
Oh, I was not even attempting to discuss using them for the code itself in my above comment. Code that's checked into a repo has a much more complex set of tradeoffs that I'm much less convinced have a one-size-fits-all answer. My point was more that even if reasonable people can disagree on how much you need to care about the code itself being LLM generated, I can't wrap my head around why anyone should care whether an LLM happened to execute the `git commit` and `git push` shell commands!
> might know plenty of git, but haven't worked with Github
That seems very unlikely. Github is the entire reason git is so popular, vs. any of the other distributed VC systems.
Edit: I'll refine that, after a moment of thought. Git is popular because of Github. But, I guess that now maybe there are people who are using git because it's the popular/default version control choice, but they might not be using Github. They might be using another forge, something homegrown, or just manual builds.
I run a live coding challenge on a rather benign coding problem with the explicit expectation that candidates will use an LLM.
Interestingly, an LLM involved feels like it’s made it easier to assess candidates. Senior candidates know exactly what they’re doing and strongly driving the LLM. They pre-communicating design decisions, gotchas, and requirements. They also ask a lot of clarifying questions and dial in the scope (after all not building something is better than building it).
Less experienced candidates tend to jump right in, build way too much, and stumble because they delegated all of the critical thinking to the agent. They often end up in a place where the agent doesn’t align with their vision and they can’t really reign it back in.
Git is a simple tool. I expect any engineer can learn to use it. While git is common, I would not expect an engineer who worked at a company using a different version control system to know how to use git (which is a really awful - I still miss mercurial). I expect you to know the general concept of version control, but the specific on is a tool I expect you can learn quickly.
At one job I had, we used git extensively, but not GitHub. We had internal systems for review and branch management. I could tell you all the gory details about various workflows and landmines in git. Is the test here really about git, or git+GitHub?
I rarely use anything beyond checkout, clone, pull, commit, cherry-pick, revert, and push in a job. If I have to, I'm probably asking the AI what to do by giving it my situation. And hopefully the AI can read about whatever git standards the company would like to be enforced so the engineer doesn't have to interrupt someone for that.
This is more about opportunity cost because instead of asking about git, you could ask about other valuable skills in your limited time with the interviewee, but I guess if git is really that important, then it's justified.
Well, I've been using git at work for 6 or 7 years, but I can't out of hand open a PR; at work we very rarely use different branches and the last time I touched Github I was a student.
Do you mean specifically Github, or any code forge at all? When you say "we rarely use different branches", what do you mean - you're all pushing to master, or you have branches for each team or something? I would regard "code forge" and "separate branch per reviewable unit of work" as extremely normal, nearly universal.
> (we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
How does that work? Git has no concept of pull requests.
Something similar for senior devs: code reviews. Reading and understanding code has always been important. I find offering simple code reviews very telling. In a way, your LLM approach matches that. It's a way to do a code review. It's a test of understanding, and not just memorization.
yup. and it's incredibly annoying, specially when a new joiner is writing a PR that a more experienced employee is reviewing.
i had to argue with someone, more than once, that "cursor is wrong about this assumption because i originally wrote this" because they just accepted whatever model they used as truth.
I want to use Alacritty on Windows, but it still has a ctrl+alt+key issue due to some upstream winit issue. Other apps seemed to have worked that out in other ways tho. I thought the font rendering in Alacritty on Windows was the best, so I still hope one day its updated with a fix.
i'm using LLMs/GenAI to do one thing: write unit tests.
since i really don't follow the idea of "writing unit tests first", i implement the feature, test as a user, and then use LLMLs to write the basic unit test. then, i will write more tests to make sure i'm covering everything.
feels like an ok-ish compromise because LLMs can do some ok job with defensive code, while i maintain the main implementation and more advanced test scenarios.
honestly, one day i will understand why these guys go so hard on planning vacations.
one of the best parts of vacations is actually planning it! trying to find restaurants, organizing your days, trying to fit activities -- that makes you look forward to your time off.
really, i want to automate the boring parts of my life (did i really pay rent this month?) not the things that make me happy.
Social media has fried some people’s brains so much that it is incredibly anxiety inducing to make many typically mundane or routine choices, for fear of missing / missing out on something.
I think they're grabbing at universally-understood frictions that people associate with pleasure or anxiety … or, most powerfully, identity.
We all know the feeling of wanting to be successful, or to go on a great vacation. Showing these scenarios creates strong emotional response in the audience - we're now the people winning at work, going on a great vacation. This is especially important when you're selling something dull. See: insurance.
The best part is, they are totally able to one shot airgapping. You are too. Go ask your local agent to set up a pre-configured Linux VM with whatever stuff you want on it, then ask it to airgap it allowing only X, Y, Z services. It will one shot it.
You know what's better? They already do this per (paid) user - your ChatGPT subscription comes with a Linux VM that you can even legitimately SSH into, just ask your agent to configure it to accept your public key.
They absolutely know how to spin up VMs and configure them. They just made the conscious decision not to for the task where they specifically instructed the agents to hack stuff.
Anything that's accomplished simply by running software does not qualify as "airgapping" in my view. The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them.
But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.
> The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them
If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
> If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
Well, my mental model concludes: because they are incompetent WRT security, or at least they inappropriately trusted a third party that turned out to be incompetent.
They are not incompetent. I personally know a few people on their security team. They are world class security engineers. There's no way they didn't know.
Real air gapping isn't conceptually difficult. It's just a lot of effort and expense (although as far as I can tell, tiny compared to what's involved in building entire new data centres), and adds friction to your operation. (And you also have to have competent, trustworthy employees, as Stuxnet showed us.)
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
(edit: spelling)
reply