I work at a place that has a few very large code bases, we use trunk-based. I will never go back, "what is your git workflow?" Is a question on my list of things to ask potential employers now and the only good answer is "trunk". So much mental load is wasted on these other strategies.
It gets even worse — convoluted GitHub action templates that no one can debug, attempting to impose on every newly created repository.
The projects at my company that use the complicated actions with gitflow are slower and less efficient.
When I release a service at my company I just do trunk based ,but I have to redo the GitHub actions initially to save it from itself. Over the long run it saves me a lot of time.
Trunk based development fits well with well defined sprints and continuous integration but I'm not sure that not committing into a release branch is the best strategy when the releases are kept small and the development continues with full force after the release is formed. In my experience it makes sense to fix found bugs in the release branch and then merge them back - expectation is that these fixes are really minor as the release is not created before mainline development has been validated and is ready for the release. No new features should be added into the release branch, only bug fixes. In such case merging back small release branch changes has not proven to be any pain at all.
I'm not really sure what the mental load is but having gone to Trunk felt like a mistake, I don't know if it was just designed wrong at the last place I worked at, but we have to do more than normal per release, and I'm not a fan of squashing commits, it means I can't just go back to a branch and merge the development branch back into it, I have to check out a brand new branch. I also preferred having "master / main" and "development" because if QA claims there's a bug introduced during the development phase, you can figure out if it's already in the main production branch or if it truly was introduced during development. Which is something you can't really do with "trunk based development" it's like all these people who praise it should just switch to SVN instead?
They deleted every branch on merge, and had us rebase for merges. Before trunk based development was introduced, everyone on the team understood what the flow was. Afterwards, whichever developer was picked to cut a release, took half a day to figure it out to make sure they didn't screw up.
I would have rather just used SVN and called it a day.
Nothing is lost in git without great effort. Deleting a branch is simply removing the label from the history. Just take note of the hash of the latest commit on the "deleted" branch. It'll still be there and can be checked out in detached-head mode.
Wait, this doesn't sound like trunk-based development.
This sounds like a traditional branch-based workflow.
> to cut a release, took half a day to figure it out to make sure they didn't screw up.
Do you remember what was making the process complicated?
In trunk-based development, you ideally can tag a release wherever you're at. Your trunk is always stable and releasable.
If it's not, you fix it and then release.
GitHub "deletes" branches on merge, but it doesn't truly delete them - rather, it removes the branch, that's deleted, but the commits comprising the "pull request" are saved and referenced from that pull, so they never go away. They're slightly harder to find in GitHub and may disappear from your local copy, though.
There's a lot of other moving parts to trunk-based, they can only be taught by experience. I've written the tutorial where I have people make mistakes, and then fix them, but the problem is that they make mistakes in making mistakes.
> Wait, this doesn't sound like trunk-based development.
I'm not sure, I just know my employer had a third party managing releases for a while, then the guy who pitched Trunk based dev left, and we were left with people trying to follow the breadcrumbs.
If you deploy your main branch on every commit, there is no "development phase". You should deploy more frequently and work on smaller changes. So when your QA (or a user, or a monitoring alert) finds a bug, you are pretty sure that the bug was introduced recently.
Google and Facebook both use trunk based development for nearly all code at both companies.
This guide advises you to consider trunk based development if your release cycles are less than ~4 weeks. I'd suggest doing it no matter what. The only place where I've run into problems with this was managing very long term hold outs where we wanted to avoid changing user visible behavior. Instead of branches, we maintained literal copies of all the files for multiple years. I don't think revision control is the right place to solve that problem though, because there were no clear service boundaries to decide what things to pin, we just did it in an adhoc way.
I am amazed at how much “process” developers these days have to adhere to, including complicated branching strategies plus a whole lot more. So much time wasted doing busy work.
I have always been happiest in a trunk-based dev model, but in many orgs it is heresy to even mention it.
Imho this is an emergent property from the fact that most folks don't understand what they're doing. If they did, then every time Mr Dilbert pointy head dude comes up with some nonsense process, they'd explain why it isn't beneficial. Instead most of the time everyone nods. Lawyers don't have crazy people telling them how to write contracts. Doctors don't have people telling them how to saw off legs. Teachers kind of do have dumb people telling them how to teach though.
Lawyers do. My dad is winding down his legal career but the bulk of it was spent writing corporate loan contracts. He’d constantly have clients asking him to make “small changes” to the document that either made no sense or were, in fact, large changes that would rob him of his weekend. While his specialty was niche, it was not so niche that appeasing the client for their future business was not important. His clients drove him crazy with this stuff but at least he could bill those hours.
I know less about doctors, but they certainly have people that think they know better and question their course of treatment. Just look at the pandemic and vaccine nonsense. Doctors benefit from supply and demand (artificial or otherwise) meaning that they’re basically always booked up.
Developers have neither of those privileges unless you’re an in demand consultant.
Someone on HN loudly proclaimed that requiring a code review before merging was antithetical to "trunk based development." He shared the link in OP a dozen times as "proof", but I couldn't find anything in there suggesting such a thing. I also know both Google and FB require code reviews.
Was the guy just totally full of it? Because honestly I wouldn't touch any "strategy" with a 10 foot pole that discourages pre-merge reviews.
There didn't use to be any mention of pre-merge reviews in that website. In fact, the text in the website used to be pretty emphatic against any branching whatsoever.
I have known and worked with Steve Smith (the only "friends" mentioned in the website [0] - hi Steve! :D ) for years and we had many discussions about the merits of short-lived branches for peer review. Myself and a few others have argued that you can have pretty much all the benefits they were looking for with TBD without the draconian "always push to trunk" rule. It just takes a bit of discipline and communication, but to be honest I think the draconian one needs even more of those if it is to succeed.
I'm happy that the website has been updated since (it's been at least a few months, maybe even a couple years) to acknowledge and "allow" that strategy.
Post-merge reviews obviously help with the same goal of having very little work stuck in WIP. But it's a 80/20 kind of goal, not all or nothing. Post-merge reviews are probably only worth it when you can trust the developers to not screw up too much and to broadly agree on quality standards.
I spent quite a while trying to find the post and comments but came up empty. Am kind of hoping the same person will pop up here, as they seemed extremely passionate about pointing it out.
I think their point is more that the usable scale of teams/codebases is not limited by trunk based development. Which that is pretty ample evidence of.
I get that. But aren't we all struggling with complex branching strategies and microservices, because someone told us "that's how Facebook/Google is doing it!"? :D
I think that good tooling is a necessity to do trunk based development well - not just CI/CD and testing on PRs, but also being able to have stacks of commits in multiple PRs before landing as a whole (from ghstack, Sapling, or Graphite).
Having worked at Facebook before where managing stacks of diffs was far better than GitHub, I wish that more of the improvements to the developer workflow could be open sourced beyond Sapling now that Phabricator is no longer supported.
I agree with what you said but I'd further refine it by saying not just having testing and CI but actually having "high" test coverage and tests that actually test the code.
Which of those three tools works best on GitHub? Ideally without paying (not that I mind paying but it’s impossible to get the approval for some niche tool like this)
TFA advocates strongly against long-lived branches, which I understand and agree with, but with one exception: major/LTS versions should have their own branch, so that hotfixes can be cherry-picked from main (or, rarely, applied directly to the major version branch.
That's right. It's important to be able to reproduce released builds with minor patches, and branches are the right tool for that job. But resist the temptation to fix that CVE on the release branch and then upstream it, because then you're once again doing branch development, treating trunk like a garbage can.
Nice to see this getting some treatment. It's been my preferred source control strategy at several places now -- both small and large, though nothing so large as Facebook.
It has its limitations, but for the the 90+% of places that aren't operating at Facebook scale, it's probably the right choice. It doesn't eliminate merge hell, but it does localize it, making it a lot easier to resolve conflicts without accidentally undoing somebody else's work.
It's especially well-suited for any projects that are in constant-iteration mode, where there's a steady stream of bugfixes and new developments getting deployed daily (or even multiple times a day).
You can even manage longer-lived branches after a fashion: the cleanest pattern seems to be to capture your changes into a patch, revert your branch to the commit immediately before your changes, merge/rebase main, then apply your patch on top. This is bad and lame because it rewrites history and requires a force push to get it into the remote, but it's also nice because it keeps your changes at the tip of the branch and makes review a bit easier in e.g. GitLab.
Every small shop I've worked with that has tried to use more complex branching models eventually hoses the repo and ends up burning a lot of dev hours one day trying to make sense of a git log that looks like a diagram of network sorting.
A lot of the rebase/merge hell I've seen has been more or less resolved with the introduction of a merge queue. Granted, there are other issues that crop up (notably, flaky tests go from being an annoyance to life-threatening) but at least bigger changes don't get blocked and stay blocked.
I'm not sure if any branching strategy helps supporting multiple software versions simultaneously. There is a reason why "rolling releases" are the default now.
For a short while it may work to just merge bugfixes to older versions. But after a while the changes to older versions need to be done independently. Implementing a fix for three versions will just take three times longer than for one version very soon.
Could use a (2015) - according to the front-matter on the repo (in the master branch, natch). It's been posted here a number of times, it was particularly popular the first posted time in 2017[0] (199 points, 208 comments).
I’ve been pretty happy with trunk-based development in my job. It’s simple and reliable. I don’t miss alternatives in any way.
A significant amount of Git-isms are based around Linux kernel needs. Which is great. But closed source and typical open source needs are pretty different, and much simpler.
This site has developed a lot of content without seeming to directly address the major concerns anyone coming from git flow will have. For example, their code review page does not really address how github and gitlab are both set up to support branch based review, and trunk based review is a lot harder when using those tools. Similarly, I poked around and figured out what they recommend for hotfixes, which is just fix it on trunk and cherry-pick the commit. That works sometimes, but anywhere with infrequent releases (like regulated environments or shipped software) may find that pretty painful. I suspect I could raise more issues if I put more energy into it, but it's hard to want to, given how the content is presented.
Overall, this site reads like someone who's so convinced they're right that they don't bother to take other viewpoints seriously enough to actually refute them.
We’re considering this at my org. We’ve got a distributed backend with about 2 dozen repos, with a big web of dependencies between them.
When you need to make a breaking change to one that is depended on by a lot of the other repos, it can be incredibly tedious to fix all the downstream effects.
Using trunk would resolve it, but would also of course introduce other issues.
Multi-repo software products maintained by the same team are another thing I'm probably never going to understand. In reality the components are never fully decoupled and nearly always released together.
When splitting up a product into multiple repositories a lot of new problems are created. Every sub-project needs to be testable completely independently and then in a later stage integrated with the other components and tested again.
With multi-repository it's much harder to isolate a change over multiple components into one PR and test those changes together. Instead they develop separately and after some time many of those changes are tested together. It's just making the feedback loop longer without any added value.
> When splitting up a product into multiple repositories a lot of new problems are created. Every sub-project needs to be testable completely independently and then in a later stage integrated with the other components and tested again.
While I'm a fan of monorepos, that issue doesn't completely disappear there, does it? I mean, if the monorepo contains the code for N artifacts (say, N, microservices that interact with one another), you will still have to make sure that the master version of microservice 1 plays nicely with the currently deployed version of microservice 2.
I think this kind of complexity is not the reality of most developers. They just stop all old services, update them and re-start them during a maintenance window.
That'd be news to me. At least in all big companies that I've come across, where revenue depends on services to be available 24/7 and maintenance windows maybe happen once a year, I have never heard of such a thing.
How does this look like when you have multiple active versions of an application? In some cases there are apps deployed in different environments (or to different clients) in different versions and having a single place to make all commits is not feasible.
How do things that have LTS versions in parallel with "regular" versions organize?
How I've done it: Just tag the commit that is used to build a release. If you need to add a fix to an older release, create a branch from the tag. Do the changes trunk based on this branch, then tag and release. If needed just repeat it.
I'm convinced that complex branching strategies never magically solve any issues. They just confuse many people until every successful release feels like dark magic.
I don't like long lived branches. For me a branch is work in progress. Work should be finished, once finished the branch should be deleted. If you start work again, create a new branch.
But sure, if supporting an older release is some kind of ongoing trunk-based process this is also a viable solution.
There's a workflow where you commit to each release branch?
I genuinely did not know that. I thought working on individual branches, merging to master, and then cutting the release branch from there (to merge back into master if any changes were found during testing) was the only major way people actually did things.
Your local repo is the equivalent of the development branch. Yes, that means you don't collaborate with others on development branches, or go for weeks committing only to a branch.
If you save up a bunch of work, dump it on the main branch, then start a new development branch that lets you escape from the integration pain of your big merge, then trunk becomes a dumping ground that nobody wants to (or can) work on.
Seems strange. Sometimes we’ll have feature branches that are made up of different parts, with different devs collaborating to build those parts out. For this I don’t see a way around having some staging branch with two devs sending PRs to before you work out all kinks and merge the staging branch back into main
This is the hardest part about trunk based development - changing your way of thinking. Everything needs to be decomposed into much smaller changes, and you need to think about the impact of each of them being deployed into production (since that will happen). New features should exist behind some kind of feature gating or dial-up capability, or a new API version with restricted access, etc.
That seems painful but it's less painful than merge hell or deploying a change with a massive delta to production and needing to roll it back and unpick what went wrong.
"before you work out all kinks and merge the staging branch back into main"
Working out all kinks on an isolated branch is very hard to do. You need to integrate to hear the screams of the team you never expected your change would affect. So your choice is to integrate commit by commit on trunk, or to wad it all up into a single waterfall release from your branch to trunk, and then spend hours/days bisecting to find which commit caused the screams.
As jozzas said, feature flags enable your code to safely lurk on trunk before it's functional.
So if the “deployed branch” were really just a “tag”, that would fit trunk based development? Because that’s basically what git is doing. It’s not like there are two sets of files anywhere, afaik.
My understanding of this though is that there isn’t supposed to be a difference between “what we’re working on” and “what’s deployed”, which I think then means you don’t really do “long term” development.
I've been working on my product for over 8 years, sometimes alone, sometimes with a tiny team.
I have a mono-repo with one master branch - that auto-deploys to test - and one release branch that auto-deploys to staging.
I can hot-swap my staging and production environment with the click of a button.
Simple, and it allows us to deploy a hotfix of needed, and we've done some major work on master that needs to be validated first...
But,... It all depends on your context and your needs...
Sounds neat, but I still think complexity is removed if the deployment is not directly linked to the code repository. For me the code repositories purpose is to build software, and operation is not it's responsibility.
I prefer having a gitops repository that describes the environment. Completely decoupled from the source code repository. If some automation directly from the source repository is required, a CI job can automatically update the gitops repository and trigger a deployment.
I'm convinced that development and operation (deployment) should be thought of separately. The branching model for the code should only serve the purpose to create a working product (artifacts of some kind). Those artifacts should be versioned (can be semver, or just the short hash of the commit, or anything else).
Once something is ready for deployment it can be deployed, in the best case this is just clicking one button or one CLI command.
Developers shouldn't be afraid that one wrong commit can automatically just go to production.
I think the idea is that, rather than storing two versions of the code in your version control system, you store both versions in trunk and gate with a feature flag.
I've worked on teams using trunk based development before and I ended up being the one having to fix other people's messes, deal with them losing work, and the absolute hell of trying to find where a bug was introduced.
The only people I really want on my team are those that understand Git. And even when there are team members that don't understand Git, I sure do and can pretty easily un-eff any situation someone gets themselves into.
Seriously, Git is actually simple and it should be a fundamental CS class taught at all universities.
Yeah data is often recoverable by experts. But it’s too hard.
My VCS rule is that commit ids are sacrosanct. If you make a commit that commit is safe. You can do whatever the hell you want. Merge, rebase, whatever. But you can always go back to that commit.
This is kinda sorta true for Git. But that isn’t a first principle. And Git’s management of branch labels is super finicky.
Mercurials anonymous branching is a better default imho.
I see this as typical of organizations that value creating a more inclusive environment for their developers.
Also, for what it's worth, Jenkins changed their terminology to use "agent" instead of slave in 2020 [0]. These efforts might seem futile in isolation, but they add up over time.
The comment I'd make is that this appears to have been settled nearly eight years ago in this particular context, and that maybe you should give it a rest.
It was "settled" in that we're all agreed it's obviously incredibly stupid but not worth fighting over and getting called racist.
The good news is that "master vs mainline" was so embarrassing that even the people who push this stuff realized they overreached and have settled down since.
