I've got a much nicer branching model- try not to have one. Everyone works off master, and you aren't allowed to check in code that won't run in production. Hide unfinished features behind feature flags, and never merge/push a change that won't pass tests/CI.
The chaos of huge feature merges (a key source of bugs I've experienced) is minimized. You deploy fixes hourly, not weekly (or later monthly when it just won't seem to pass CI). The time between code being written and a bug being seen can be reduced to minutes and hours, making finding the root cause a breeze.
I think from the perspective of the central repository, this is a sane approach. Locally though, you should still be routinely using (short-lived) feature branches. Working directly off master on your local machine can make things annoying / complicated when you need to move away from your work for a moment (say to fix a bug) and don't have a good representation of "what's currently on production".
Also, I think this approach necessitates that you have some form of continuous deployment, or strong assurances that master is almost always on production. If, for example, you have a fixed 2-week cycle for production releases, "just work off master" will be a source of pain when you inevitably have to fix a bug on production without rolling out the other commits that are there for the "next release". That might not be an optimal way of developing, but it's about more than just how you use git - it's really more of a delivery question.
Depending on the frequency of your deploys you can simply add release branches or tags to the “work on master” model. Any decent deployment system can deploy from an arbitrary arbitrary ref, plus release history in your git log is very useful. The point here is to eliminate long lived feature branches that require their own maintenance.
A hotfix branch fixes this. You can still do trunk based development for the rest, and cherry pick the bugfixes into such a hotfix branch (branches off from the latest commit that was released).
No it didn't. You're seeing things black and white just to win an argument. It advocated against feature branches, git flow and all that. Occasional never-merged-back hotfix branches really don't undermine the core point.
Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there. Some concrete questions:
- How do you prevent your codebase from becoming if-statement spaghetti?
- How do you prevent new features from being leaked to the user? They'll see the new features in the front end source code. At many companies this isn't an acceptable tradeoff. So do you preprocess the release code and strip out the disabled feature flags? With what?
- What do you use to control feature flags? Just a json file filled with `"foo-bar-feature": true/false`, or something more sophisticated like a control panel that you can use to say "10% of our users will see this feature"?
>Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them
I worked at Etsy from early 2012 to late 2015 and I used feature flags every day I was there.
>How do you prevent your codebase from becoming if-statement spaghetti?
You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go.
>How do you prevent new features from being leaked to the user?
You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns.
>What do you use to control feature flags?
You can use a service like LaunchDarkly or Optimizely or you can write your own service, or you can have a file in memory. Usually it starts simple with boolean toggles and evolves into a ramp up system that let's you selectively target users. The important part is that you need to be able to change features without redeploying code.
You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go.
I don't understand. I was asking about the development source code, not the code delivered to the user. The dev source code isn't stripped of if-statements, right? That's where the feature flag toggling lives. But then it seems like it's easy to devolve into spaghetti.
You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns.
Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this?
I looked up LaunchDarkly. $299/mo for basic team support. Uhh... That's almost the cost of WeWork office space.
Are feature flags really so nascent that there isn't a FOSS solution for it?
> The dev source code isn't stripped of if-statements, right?
When I use a feature flag as an alternative to branching, then I delete the if statements once the feature becomes permananent and gets released to all users. It's analogous to merging a branch. Obviously I'd leave it there during QA and A/B testing, but once toggling is no longer necessary, I remove the toggle.
> Ok, but how? Assume an express server.
You probably have feature flags already and just didn't call them that. But normally in any given project, there are multiple ways to wrap a feature in a toggle.
Here are a couple of examples I use:
- I use handlebars for my frontend templating. When I have a feature that needs to serve or withhold some HTML to/from the user, I put the feature flag in my JSON file of variables that get passed to handlebars. Handlebars has a way to wrap HTML with a boolean test.
- I use the Google Closure compiler to optimize, uglify, and minify my code. When I need a feature flag that is code only, and I don't want users to accidentally get exposed, I use a boolean @define in JavaScript, and wrap feature code in an if statement. When the Boolean is false, Closure compiler removes that block from the output code.
- I set up my own JSON file of config constants that are injected into the code during build. If I have a feature flag to A/B test something, or that I do want released to users but has a secret switch to enable, then I put the constant in the JSON file, and write UI code to toggle the flag. This one usually generates more code than the two above, but whenever I decide that the feature is either dead or permanent, I remove all the code to enable and disable the feature.
- For fully dynamic toggling, I put my boolean config variable into my database. I use Firebase which is a giant pub-sub, so I wrap the toggle in a subscription to the database variable, which might then (for example) inject some HTML and/or toggle the CSS display attribute of some things on my page.
> Are feature flags really so nascent that there isn't a FOSS solution for it?
No, feature flags have always been around, since long before branching. There isn't a FOSS solution for it separate from your framework(s) because how they work depends entirely on your stack. They're also normally really simple, adding the dependencies of another project just to have a boolean in your config file or in your code is usually overkill. That said, there are FOSS and commercial solutions for A/B testing, which is usually a feature flag workflow, but it comes with lots of other stuff you might not need, like the reporting, analytics and statistics parts.
> I delete the if statements once the feature becomes permananent
Right. In theory. Does this really scale when there's more than a dozen developers touching the same code paths? It's not just you scratching off your own TODOs, it's multiple entagled condiditionals. Can you really delete your flags with confidence that this does not affect any other code path?
Which brings me to my real question: How you you guys test this stuff? When I have experimented with feature flags before this has been to much detriment for testing. Suddenly the new payment API our provider will launch in June needs to be tested with the new backend we'll launching Tuesday, and with the new asynchronous database driver our DBAs wants us to migrate to. And so on. The complexity explodes. Especially when the features have dependencies external to you.
When everything is in master and everything must be in shippable condition you need to test everything with everything else, while previously the new payment API in June could live in it's own branch and test on their own schedule and not bother everyone else everytime the payment provider's test environment took a nosedive for some reason.
So you either end up with brittle tests that can take days to execute, or you end up scratching a lot of the less pressing combinations. Which brings us to the situation above. How can you then be confident to remove conditionals without testing the newly exposed code paths?
It is so far my suspicion that most people who speak of feature flags as an alternative to branches either work with trivial products where most developers fit in a small room (in which you are not the use case for these things) or talk out of some theoretical conviction which has yet to meet with reality. These things are hard, because of all those details which seem small if you squint enough.
To be clear, I personally use branching a lot (e.g., for sprint-sized features) and I create new feature flags rather infrequently (for multi-sprint sized features or for A/B tests), and only for pretty large features. But I can see reasons why some teams/environments do go, or want to go, feature-flag always. Before git, feature toggles were more widespread as a workflow.
As far an entanglement with other code, I would evaluate this wrt branching. If it's entangled, the branching analogy is a merge conflict. The most common case is you can delete the toggle without conflict, and occasionally you have to spend ten or twenty minutes untangling. Every once in a rare while it gets messier.
For testing, the prod tests run the prod config, which is likely with all optional feature toggles turned off. My team would run our own tests with our feature toggles turned on, but not with other teams' feature toggles. I don't run all combinations of features. Usually, a feature toggle will also wrap new tests that belong to that feature.
Again, the same question you ask applies to a branch workflow. Do you run all possible branches at the same time? Is it an exponential combinatorics problem to have multiple branches? Not really, each team runs the tests for the features they work on. Same either way.
The great thing about feature flags for testing is that you have greater control over who is exposed to your new feature. With a feature toggle, you can expose some teams in your company outside of your own, without exposing the whole company. You can expose 10% of your customers and do testing without inflicting bugs you didn't forsee on 100% of your customers. If there's an emergency, you can limit the damage radius with a well designed feature flag. Branches can do this only to the extent that the merge structure allows it, and they can't help with customer testing.
> It is so far my suspicion that most people who speak of feature flags as an alternative to branches either work with trivial products [...] or talk out of some theoretical conviction which has yet to meet with reality
I think I know why you feel that way, but that sounds (to me) like you haven't been exposed to (and so can't imagine) workflows outside of git branching. I mean no insult, but you are jumping to conclusions from lack of experience. Feature toggles as a pseudo-branching workflow predate git by decades, they are used on the largest codebases in the world -- did you miss the comments in this thread from MS and Google?
Feature flags are used in nearly all repos where branching exists. It's not one or the other, you can use both workflows at the same time. A/B testing is based on feature toggles, and all sizable websites on the web are doing A/B testing constantly. Chances are high that you use them already and just haven't yet recognized that they're the same thing we're talking about. If you have config files for your project, then you have feature toggles. If you ever change a value in one of them, then you could instead branch your code, remove the config variable and rewrite the code that used to reference the feature flag to be hard coded instead. Is that making sense?
FWIW, just to put some specifics and back up my own 'conviction' with real world examples, I've been on teams that used feature toggles to test and ship features during the production of several animated films, movies you probably saw. When I worked in film, it predated git. We had a hierarchical repo, but there were no branches, only pushing to master and feature flags.
When I worked in games, the team I lead used feature flags to support cross-team testing on a $1B console franchise.
At a certain web company, in addition to constant A/B testing, my team shipped features to customers who signed up to be on the 'beta' versions of features before they were rolled out to all of the >1M users. We used feature flags to control which teams internally could see & test certain features, and we used feature flags for either long-running or large cross-team features when branching & merging would cause undue amount of branch noise.
And at my own startup now, I prefer to wrap my features in feature flags even when I'm branching. That way I can back out a feature when it causes problems or has bugs without having to do git surgery or even risk a merge conflict. With a few thousand active customers, this has saved my ass multiple times.
> If it's entangled, the branching analogy is a merge conflict.
I like this observation. As long as two developers touch the same code, you will have a conflict. Now your choice will be where to best place that conflict. In an SCM, in conditional statements, or somewhere else.
> For testing, the prod tests run the prod config, which is likely with all optional feature toggles turned off.
I don't understand this. If you are going to activate a new feature in prod, surely you test it first? Or do you start testing all over again before you switch on a new codepath? Feature toggles must also include externalities such as backend systems changing their APIs. Even if you are forwards compatible, you must take care so your application doesn't break when that happens, and that means testing in advance.
If you really follow the master-is-production branch free model all merging must cease before you activate a feature in prod? That doesn't sound very practical.
> Do you run all possible branches at the same time? Is it an exponential combinatorics problem to have multiple branches? Not really, each team runs the tests for the features they work on.
A team can not be responsible for testing only their own development work, they must also know that it works with other people's changes that might activate before theirs does. That cross team communication is hard enough using branches, but at least teams do not disturb each other's testing. That enables asynchronous development. In a master must always be production model, which seems to be what most branchless development people seems to advocate, every combination is blocking. That's the difference.
> The great thing about feature flags for testing is that you have greater control over who is exposed to your new feature.
Right. I'm not saying that feature flags are useless. I'm just questioning that it can be used instead of branching in non-trivial projects. There are a lot of use cases where it makes sense, for example user interface stuff where branching unnecessary.
As you point out, feature flags are just a new term for application configuration. We've had that forever. It is as an alternative to developing in branches I question it. We generally don't celebrate making everything an option.
> I mean no insult, but you are jumping to conclusions from lack of experience. Feature toggles as a pseudo-branching workflow predate git by decades, they are used on the largest codebases in the world -- did you miss the comments in this thread from MS and Google?
Oh, absolutely. I type this because I want to hear from people with experience. What I meant by "my suspicion so far" is that the experience with branchless development I've heard so far is either trivial or theoretical. There's a lot of pointing at Facebook as a good example but not much in way of first hand knowledge.
And yes, I did notice the comments. The Microsoftie called it "if-statement hell", the Google commenter said it "impacted developer velocity" primarily because of the testing requirements. The former can perhaps be explained with culture, but the latter matches my experience exactly. These comments concerned feature flags in general however, not branchless development which is a special case of it.
Your examples from the game and film industry sounds like it could be made workable, but I imagine that development looks different from the average ERP system or backend service. Certainly with smaller startups you shouldn't overcomplicate matters.
> If you are going to activate a new feature in prod, surely you test it first?
Yes, my teams test features before pushing to master regardless of branch strategy. What I meant is that the prod/master/nightly tests stil run the prod config as-is. Teams run tests, but features under toggles aren't tested using the master branch's automation tests until the feature is turned on. Exactly like how a team branch won't get tested in the nightly run until merged.
The QA dept. is typically running the staging branch, which is whatever got pushed to master, but hasn't gone live yet. When I turn on a toggle, it's just like I merged a branch, so QA starts testing the new features at that point. When we want extra testing, we set the feature toggle so that QA is exposed and the public is not, then if it passes, the code goes live with the feature toggle turned off.
> A team can not be responsible for testing only their own development work
When we test feature toggled code, my team runs all the automation tests with my team's features turned on. Similar to how we'd run tests as if our branch was merged into master, but without pushing the merge.
What we don't do is speculatively run tests with other teams' feature toggles turned on. Exactly like how you wouldn't run tests against someone else's unmerged branch. Their features are tested against ours after one of the teams turns their own toggle on, once one of the features is made public -- to at least our teams if not everyone.
> The Microsoftie called it "if-statement" hell
I've heard first hand accounts about their branch hell too. And it's a worse hell than I've ever seen personally. From about 10 years ago, not today, so I don't know how different it is, but someone in Research said it took a full month of merging and conflict resolution for a code merge in any given dept. to propagate to the rest of the company.
If that's happening, feature toggles definitely won't fix it. There is a chance it could make things worse.
I'd tend to agree if you strictly feature-flagged all commits as a complete replacement to branching, it would probably be ugly. Git is useful, and branches are a nice workflow for most dev.
My criteria is to choose the one that makes sense. Feature toggles are currently better IMO for long-running feature dev or very large features, branches are better for medium sized features. Adding, merging, then removing a feature toggle during a single sprint is more overhead than it's worth.
What we did in games before git was everyone pushed to master at all times (or whatever they called it in Perforce, or the lovely SourceSafe before that.) Most people did not wrap their features in toggles. We lost a ton of developer velocity due to someone checking in a bug that broke the build for everyone who pulled after that. More often than not people would hurry to try and fix their bug rather than revert, occasionally checking in further bugs. Company-wide breakage was a weekly occurrence. Those were dark days, it was ridiculous. Feature toggles were the only safety mechanism we had, which is maybe why I see them in such a positive light.
I would not recommend going whole hog with feature toggles and eschewing branches. But I do recommend trying one or two and feeling out exactly how the whole process would work for you from start to finish. They are still useful in a branch environment, especially for features that would cause lots and lots of branch noise.
> then I delete the if statements once the feature becomes permanent and gets released to all users
Doesn't this throw out some of the point of feature flags? That if your production servers are getting hammered, and you're having trouble scaling (for whatever reason), that you can degrade service by toggling off various features which are more resource-intensive so that your service doesn't crash entirely.
Feature toggles implemented through conditional logic seems like an anti-pattern to me. Developers make mistakes, and it's very easy to forget to wrap work around the necessary toggles and accidentally expose not-ready code in production, which may also have security consequences.
It seems to me like a better pattern would be to feature-toggle on an API level, and force developers to plan and commit to relatively stable APIs when they deploy new features to production. This is achieved through an API gateway model - a versioned API is introduced to the gateway, and new versions of services must pass integration testing proving that they adhere to that API before the API gateway will serve the new version of the service with traffic. Developers may introduce new code, and new unstable APIs, and deploy them to production, but they will not be user-facing until a new, stable API is pushed to the API gateway to serve to users. This preserves developer flexibility in pushing new work to production, without exposing that work to the user, until the feature is stabilized and ready.
> Doesn't this throw out some of the point of feature flags?
Not if you're using feature flags as an alternative to branching, nor if your feature flag is intentionally temporary. A branch is temporary, and merging the branch doesn't eliminate the point of branches, right?
For performance toggles you want to keep and use, then definitely just keep those.
> Feature toggles implemented through conditional logic seems like an anti-pattern to me. Developers make mistakes, and it's very easy to forget to wrap work around the necessary toggles
This is a valid concern generally, but in my experience doesn't detract much from the reasons to use feature toggles. In particular, the good examples you brought up (performance features & API feature toggles) you can't have without conditional logic, so there isn't any choice or alternative, right?
> It seems to me like a better pattern would be to feature-toggle on an API level
That's a good idea when the feature in question is an API feature. Are you a backend kind of person? ;) Lots of features are frontend, lots of features are full-stack. It all just depends on which feature, but generally speaking having API level feature toggles in your toolbelt is a great idea.
The expectation of a bunch of weird branches making everything confusing (I had that too) ends up being unfounded. You typically only see a few places with branching logic, and people usually draw attention to it being branching logic for a feature or experiment.
How and where you branch really depends on the feature. If you're A/B testing a button color, you'll deploy a different strategy than if you're trying to introduce new logic with minimal regressions. But you can plan out your strategy in both cases to minimize leaking information and disrupting user experience (and you should).
>Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this?
The general flow is: get status of features from source of truth. Check current feature flag's status. Do something based on that status. You can make it as complicated or simple as you like. You can have a global boolean flag for all features where it's always either true or false. You can have different flags for different users, you can have certain percentages of users have different values for flags. But it all starts from getting the value of a flag for this current request from the source of truth.
Let's say you're writing a crud app on an express server that communicates via http to a database service. Your database service is rolling out v5 of their api, and they're at the last stages of their beta so they're pretty confident. You decide you want to test it by routing half of all the traffic to your crud app through v5 and the other half as normal through v4. You could lay out your code in several ways, and you'll have to figure out what works best for your model and your team.
Let's assume you have a feature service you query on every request (maybe it's express middleware) to see if an experiment is enabled for the current user, and you've queried your current user against api v5, and you have a boolean variable indicating whether to show v5 or v4 and it's used in a route handler for your home route.
You could change the url you hit based on the state of the experiment, provided the interface to the api is the same.
> Let's say you're writing a crud app on an express server that communicates via http to a database service.
As a side note - a 'database service' that's only job is to serve as an HTTP wrapper over DB seems like an anti-pattern. Service boundaries here are not related to any business concern (if you're thinking along the SOA lines) and rewriting already existing DB remote interfaces via http seems like overengineering (if you're doing n-tier architecture).
Some additional reasoning on the matter by Udi Dahan: https://vimeo.com/113515335
Would be interesting to hear your thoughts if you think otherwise.
The project I work on at Microsoft uses feature flags. I see it with many Azure services as well. I can testify that it is absolutely if-statement hell, at least in the areas that rely on these feature flags. However, I will say I think 'lazy programming' is more the issue than the flags themselves. For example, isolating feature specific behavior to it's own impl and using an abstract class to share common behavior--that sort of basic stuff gets skipped in favor of 'time'.
Thankfully these are all obscured from the user for the most part.
We have a 'control panel' like thing that we used to enable/disable feature switches. It works well for its use case and let's us enable/disable with as much granularity as we want.
Hmm. I can't get into detail. But we use it for things that are mostly 'experimental' for the customer. Let's take the all-to-common TODO MVC app and say I add a caching layer between the app and the database. I would likely add a feature flag to enable or disable retrieving or storing results in the new caching layer. Or, I add a new button that lets the customer multi-select TODO tasks, that would have a feature switch, etc. The reason for this is that it lets us enable these features for customers who either insist on having them or are excited to try new features without exposing the entire customer base to the risk of these features.
The big problem with this is: you end up having 'if(feature)' type statements _everywhere_ for each of those experimental buttons or extra components or whatever. It creates considerable debt that the team has to be accountable for cleaning up as those features start to stabilize.
> Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there.
This might be just a terminology problem. Any company that does A/B testing is using feature flags, so pretty much most of the web is using them. I'd bet more or less all programmers have them in their repos too, maybe you just call them something else, or didn't think they could be called feature flags. If you have a config file, chances are high that you have feature flags already.
For tooling, I get the feeling you're thinking of feature flags as something other than your config files and code constants. But that's it: feature flags are really simple. Add a boolean const defaulted to false in your config somewhere, and wrap the code in an if statement. When you need to A/B test or release incrementally, that's when you need extra non-trivial code around features, and that's often not something someone else can provide. Turning the feature on and off requires specific knowledge of your project's stack and UI.
I think there are many different equally valid ways to use feature flags. We use feature flags to slowly ramp up the release of new features only. We have per-user feature flag settings too.
We have a small team so the number of new features that need to be gradually released is very small, and won’t make the codebase a mess of `if` statements.
We don’t really worry about feature flags leaking to users because when we start to use the feature flag, there is already a group of users being able to use the feature (usually employees because of dogfooding, and then gradually to a selected group of power users/VIPs). The frontend JS code is already minified beyond recognition (we use Google Closure compiler in advanced mode) so that’s not really a concern.
Since feature flags are per-user, you can think of them as a column of the users table in the database.
At Google every significant feature in our codebase was behind a feature flag. Pro is that you can roll out features slowly and do a statistical analysis of their impact on user behavior in a well-controlled way. Also, you can turn off a broken feature really fast without having to wait for a new deploy (especially important with native mobile apps).
The big downside was that it significantly impacted developer velocity. You have to test that everything still works with/without your feature enabled. Also, sometimes supporting the old and new way of doing things simultaneously requires a nasty hack, and when you go back in to pull out the old code after a successful deploy you don't actually bother to rearchitect things the way you ought to.
For Google they were probably inevitable, but for a small to mid-sized company I would avoid them.
There's used a ton at Google, except they're called "experiments". I believe that term comes from A/B testing, which they're still heavily used for, but they're also used for feature rollouts.
> How do you prevent your codebase from becoming if-statement spaghetti?
A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch, and launch fully or remove the feature; consider 'hiding' the ifs behind factories that build the objects that implement the different logic; also, if you have 20 features is development for the same area of your codebase, worry- you may be trying too much!
> How do you prevent new features from being leaked to the user?
I haven't had to worry about this very often just based on my projects, but there are strategies. You could be sending the user a different version of the js/html based on the feature flags (preprocessor as you said). Haven't had to do that, but woah that would be a fun little project.
> What do you use to control feature flags?
My favorite implementation was an S3 json file that effectively encoded a decision tree based on variables used. In code, we could say "here are the five variables about this request, feature object are you enabled?". By modifying that json file, you could change the features state at run time. (Note: this was not perfectly implemented/designed by me and caused a few issues when we first used it. Oops.
But you can do very quick solutions too, read a file or make an object that decides based on non-dynamic logic.
>A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch
I think a big factor that's unspoken about "trunk-based-master" development is that it fits better with "website apps" such as Facebook and Etsy. The "live website" can be thought of as a "single executable" and trunk-based mental model maps well to that. The feature-flags as a replacement for branches doesn't result in an unusable combinatorial explosion. There are minimal # of "alternate universes" of Facebook... maybe a "Facebook for internal engineers" etc. You can tame that with feature-flags.
However, for corporate development where the output is "exe" files... e.g. version 2.x of product has dependencies on a old version of a library and a version 3.x uses a totally different architecture with different conflicting dependencies... you can't model that code evolution in a clean manner with feature-flag "if" statements. If you move the "if" statements outside of the code and into preprocessors (#ifdef) or the "build" system such as cmake, you've just shifted the problem around instead of solving it. Branches instead of feature-flags are cleaner and more sane for certain domains.
Dependency injection? The conditionals are then virtual (which implementation of an interface); the same controls apply to front-end code; global dependency configuration.Though all that infrastructure and boilerplate for explicit modularity is probably only worthwhile at Facebook-scale.
With each feature in a separate file, there is no merging of features, and if you squint your eyes you can see each possible dependency configuration as a git branch, selecting certain files out of a repository, just at runtime. There are no file-level merges, only different files, and different configuration selections. (Of course, modifying code to use the interface in the first place is a file-level change.)
BTW I've been thinking about this problem in the context of computational fluid dynamics, where you want to try different scales, fluid and container initial condition profiles and boundary conditions, as well as different discretizations, schemes and variations thereof. Basically, experimental cases (a bit like "functional test" setups).
I've worked at Sabre some time ago and we used feature flags intensively.
Feature was enabled/disabled using our own configuration server (which was used to configure many other things) - it was basically a dynamic file with booleans, that we could turn on/off.
After the feature was released we removed the if statement from the code.
Who says no one is embracing them? There are tons of tools available if you check for it. But you should google for "feature toggle" libraries and not feature flags
Feature flags _are_ widely used. They generally manifest as commented out code. Like many features that migrate down from these massive codebases it's a common behaviour thats been systematised.
> I've got a much nicer branching model- try not to have one. Everyone works off master, and you aren't allowed to check in code that won't run in production. Hide unfinished features behind feature flags, and never merge/push a change that won't pass tests/CI.
You've just described a naive implementation of the Git branching model described in the article which fails to meet some basic real-world requirements experienced on all software development projects. It matches the policy of merging changes from the development branch onto the master branch but lacking any support for very basic aspects of software development such as code auditing, versioning, and continuous integration.
It's silly to let everyone work off master if master is supposed to remain stable. If master is supposed to host stable versions, someone needs to be responsible for keeping it stable. If someone is assigned the task of ensuring the master branch is kept stable then he needs to control what gets into master and what doesn't. If someone needs to control what gets into master then other developers need to commit their changes to some other branch.
Add parallel releases and couple of levels of these real-world everyday aspects of software development and you get exactly the branching model described by Vincent Driessen in his blog post.
Agreed - you can have compiling code that passes all tests but which still needs someone to review. Or a feature that you are working on that you want to merge in later.
One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)
While I do like this approach, it can cause some pains though.
It's often a race to get your (approved) merge- or pull- request in. If you miss out, then you rebase, and try again. Perhaps if your pipeline is super fast, it isn't an issue.
Another issue is that there still needs to be coordination outside of version control/CI pipeline to ensure that things are put in the 'right order', for the times when that is important.
Developers rarely push directly to master in large organizations. Instead, a CI pipeline verifies your code passes tests and then pushes to master.
There are trade-offs here around "semantic merge conflicts", but they happen rarely in practice. To avoid making this disruptive to developers, you can have another branch that marks the latest commit to pass tests. This branch should always be an ancestor of master and should only be updated by your CI pipeline.
And why not. Rebasing should be much more straightforward/common. I remember at one point I had a 'magic' command that just did a rebase the way I expected and allowed me to merge changes in easily; it quickly made its way through the group I was working with.
I like feature flags, but it's crazy to suggest that they totally mitigate your risk. Some features will end up with more lines of code hiding the feature than actually implementing it. After it's released, you then have to unwind all that code. That code churn isn't trivial, and "tests pass, ship it" isn't really a good model if you're shipping blocks of work that you don't want to expose to customers.
If your releases processes are perfect, then it's OK, but CI tends to treat release as "if the tests pass, it's perfect" and ships it to all customers. At the very least, you need to be able to compare releases that "don't change anything" to make sure you're not breaking something because some edge of your feature didn't get hidden behind the flag.
I think it really depends on what you’re developing and delivering. Master based development is great for end-user deliverables (web apps being the big one that comes to mind, many libraries could do this too). But if you’re writing a language or LTS releases I could see a pretty good case for a branching model like git-flow.
My experience with a work using gitflow is similar to yours, it was very messy and hard to manage. It was even worse that those driving it didn’t really know much about git so 95% of the commit messages were “Merged X into Y...” instead of using a more linear history.
> It was even worse that those driving it didn’t really know much about git so 95% of the commit messages were “Merged X into Y...” instead of using a more linear history.
How so? Why merge commits are bad? Are you suggesting of rebasing each topic branch or commiting directly to the master?
Merge commits are bad because they add unnecessary complexity. Your software is already complex enough. Why add complexity to your commit logs and your overall development process?
A linear history is easy to manage and easy to understand. It might be slightly worse for people committing code, but developers read several orders of magnitude more code than they write. Always optimize for reading code over writing it.
Personally, I wouldn't argue for one or the other being nicer, they each have strong use cases at different times.
I like to use feature flags for: 1- features that need to be A/B tested, 2- huge features that dev lasts for more than 1-2 weeks, 3- features that take more than a small handful of people to either write or test.
I like to use branches for: 1- code that definitely should not be inflicted on other teams until ready, 2- features that take between 3 days and a week or two, relatively short cycle, and 3- features that need more than one person and/or more than one commit to finish, but less than maybe half a dozen.
Assuming it doesn't impact the code review process, I like to commit to master directly, without a feature flag, for: 1- any small features contained in a single commit written by a single person, and 2- hot patches and bug fixes that roll in after merge.
Here's the caveats I have with that, although I do trend towards only having a master central branch whenever possible:
1) Small, incremental commits tend to be way more reviewable than big "finished" blobs of work. Local/small branches prevent me from getting blocked on waiting for review, and allows me to continue working, although you don't exactly need a "branching model" for it.
2) Tests/CI are insufficient. For games, you'll want QA hammering on a release for days, weeks, or months with only the most conservative changes applied, in an effort to shake out any remaining weird heisenbugs. To do otherwise invites the specter of failing certification, delaying your release, or burning some pretty bad bugs to hundreds of thousands of physical disks. Do you, effectively, shut down the studio when you've dialed up the stability demands that much - or do you branch so work can continue?
The worst case consequences are no longer quite so bad as burning bugs into unpatchable ROMs, but they can still be pretty bad.
> The time between code being written and a bug being seen can be reduced to minutes and hours, making finding the root cause a breeze.
This is, unfortunately, merely the ideal happy path. Great when it happens, but fails to account for the worst case.
It can take weeks, months, to find the heisenbugs that slip through CI/tests/initial QA. If you have a CI/testing setup that can catch, say, a title exit(3)ing when the charm bar is opened for more than 10 seconds, but only on the main menu and without a debugger attached - without writing a test to catch that extremely specific edge case once you have the benefit of hindsight - I'm begging you to share! Just nailing down the exact repro steps took days.
When you have no relevant callstack (even when you do eventually figure out how to dump the exit(3) event), no relevant logs, the fundamental bug resides in third party code you don't have the source code to, and fully rebuilding takes hours because you're on a large C++ codebase with even larger asset building requirements - well, for me, it took me a week or two to root cause (or more accurately, get fed up to the point that I spent a day manually bisecting version history to track down a totally unrelated and innocuous looking changelist that ultimately made us hit the bug.)
In the weeks of turnaround time fixing such a bug, a sufficiently active dev branch will have acquired another. A release branch won't. Bam, branching model.
3) Plenty of codebase-wide refactoring is hard or impossible to feature-flag.
Tests/CI are insufficient. For games, you'll want QA hammering on a release for days, weeks, or months with only the most conservative changes applied, in an effort to shake out any remaining weird heisenbugs. To do otherwise invites the specter of failing certification, delaying your release, or burning some pretty bad bugs to hundreds of thousands of physical disks.
Eh.. Game studios aren't generally still in the dark ages, are they? Physical disks went the way of the dodo. (Every game now basically requires updates on first use, right? So even if consumers get a disc[o ball], they won't really be affected by a bug burned to it.)
> Eh.. Game studios aren't generally still in the dark ages, are they? Physical disks went the way of the dodo.
I wish! Turns out enough of the world still has terrible enough internet for sneakernet to still have it's advantages. It's not the game studios that are in the dark ages ;)
> Every game now basically requires updates on first use, right?
Sadly. It's a pretty terrible experience. Ideally it's a small optional patch, or maybe only required for online play - but your ability to do that depends in part on how bad the bugs are without it. The ability to do day one patches doesn't translate into a rubber stamp either - it's easier to get a waiver on a few heisenbugs than a truckload. And this is in the "okay okay, we'll backpedal on requiring our console to be always online" world of console dev - I imagine mandatory day 1 patches are even worse on handhelds.
And if you don't want that day 1 patch to introduce more bugs than it fixes (souring your launch, reviews, and ultimately sales) - or a release delay messing up your marketing plans (to potentially similar effect and/or extra costs) - you still need as stable a build as you can get by some hard cutoff X, so you're back to having a release branch, freezing master, or some combination thereof.
> and never merge/push a change that won't pass tests/CI.
This can be enforced as well. For example using Gerrit + some CI, you can allow people to submit a branch to be merged, but it's up to the CI to merge it if it passes all checks. There's no "aren't allowed" anymore. (Which is great because people will make mistakes)
One thing to keep in mind though is you're describing a system that works for a continuously deployed web service for example. Once you have a product with versions, you need to have a system for previous releases, backporting patches, security/point updates, etc. It's a completely different game at that point.
I also agree. Realizing there can't be perfect model, I started promoting working directly with the master.
This requires to work out the problems in advance instead of letting them wait until merge time, but this . Having one branch means every team member is more or less aware of what is happening. It is much easier and it makes much more sense to set up automated pipelines if you have just one branch. Having everybody work on the same branch seems to focus people on gradual improving the codebase instead of making large rewrites.
Exactly. I've never understood the obsession with minor branching. Actually now we never ship software in a box major branches typically aren't needed either. The insanity goes back way before git but git seems to have driven it to "11".
I've been trying to think more about this model. How do people perform code reviews with incomplete features? I feel like you lose context as you review incremental commits that span across multiple days.
The advice here given to avoid using "master" branch for development, and advice to create non-default branch named "develop" (or variations thereof) is quite harmful.
If you must have a "release" branch or "stable" branch, ok, go for it, but leave the "master" for developing. Why? Strive to have sane defaults.
Frankly, the idea that somebody must check out some extra special branch after cloning repo in order to start properly developing, is not sane.
IMHO do developing always on topic branches, never on master.
This keeps master available for fully-working software, such as for the most-recent successful build using continuous integration, or for always-available deployment, or for external users, etc.
To create topic branches, here are git alias commands that you can customize as you like for your git workflows:
What's the point of running CI on a branch that's effectively for tracking releases only?
We run CI on master branch and fix any build breaks or regressions immediately. (That's the job of unit tests and integration tests.)
In fact, if your team is smaller, you do release from master branch directly, just tag the release revisions. Then if you need to hotfix it, check out the tag, make a branch, cherrypick a fix to the branch, and do point release from the branch: problem solved.
That's if the client running the (older) release version is large enough to justify releasing hotfix instead of doing fix to main stream and just asking to upgrade to latest release.
Yes to topic branches. Yes to "cactus model" of rebasing the topic to master. No to the idea that you need extra long-lived "develop" branch in parallel to master.
> What's the point of running CI on a branch that's effectively for tracking releases only?
I'm recommending using master as the output of a successful continuous integration, and thus always ready for release.
I'm rejecting any git flow that has any developer pushing any code directly to master, while hoping/guessing that master branch integration will succeed.
Caveat: Bugs will still happen.
Caveat: There are more-advanced release processes such as blue/green, alpha/beta, canary/throttle, etc.
Caveat: There are more-sophisticated integration techniques such as an internal private master branch that differs from an external public master branch. YMMV. Use the right tools for the job.)
Caveat: There are more-advanced release processes such as blue/green, alpha/beta, canary/throttle, etc.
What does Facebook do? It's hard to imagine that they slow themselves down this much. They tend to hide features behind feature flags, but is it known what their CI process is like?
Github pushes topic branches to production and if they don't cause troubles then merges to master. The idea is that good state is defined as works in production not just passed CI tests.
> There are more-advanced release processes such as blue/green
Blue/green is actually very similar to the Git branching model of forking release branches off of the development branch to merge them in the master branch, if not precisely the same thing referred by different names.
I was researching some of this a few weeks ago, and there are many posts about tags being a bad idea. The arguments are that they have to be maintained separately and they lack context.
Speaking of bad ideas, anyone want to weigh in on merge commits? I've seen arguments in favor of using `git rebase` everywhere, both ways, and to never squash commits. This makes `git bisect` usable, since you never run into a situation where it points to a massive commit as the problem.
On the other hand, that seems pretty terrible from a `git log` perspective, since many commits are WIP. But maybe it's not a big deal. My bigger concern is that merge commits provide real context: whenever you merge a topic branch into master, it seems to make sense to have a merge commit for that entire operation. But wouldn't that cause `git bisect` to always point to that merge commit rather than one of the smaller commits?
I always advocate workflows where merges never happen.
To me, the 'git log' of a master branch is like a history book about the repository. When you read through it, it should give you answers to the questions "what was changed, why, and when" in as clear format as possible.
Now, I've read most of the arguments trying to show that merges are the way to do just this, instead of rewriting history with interactive rebasing. I won't repeat them all here, but just want to ask this: when one reads a real book about real world history, does it look more like git log history of a repo which has been using rebase, or merge?
I'm not sold on the idea of the master log as a readable narrative.
If you really want to maintain such a narrative, it would be possible to do it separately, in something like a changelog.
You might even hire someone to do that, a kind of technical historian. It's real work, because software development is pretty messy.
If you reaaally care about the history of the software's development, I would seriously consider aggressive rebasing of the repository even much later -- you could refactor the commit history as much as you want to clarify the logical progression of the software.
As I see it, a source code repository is not like a history book, because a history book is written after the fact by a trained historian who spends a lot of energy on tidying up the narrative and making it actually comprehensible.
A source code repository looks to me more like an archaelogical artifact with some terse notes sprinkled in there as clues by the various workers.
Basically I think the git log structure is kind of overblown and workflow arguments that hinge on the legibility of the repository's graph structure don't really matter that much to me.
I still sometimes write pretty involved commit messages, but that's a kind of separate issue from these "workflow" discussions that are mostly about how you should formally arrange the DAG. And I also know that my commit messages are mostly lost in time like tears in rain, so I try to communicate important changes in other ways.
It looks more like merge. Chinese history and European history merged around 1200, and then again a few centuries later. American history and European history merged around 900 and then again in 1492.
But, why not both? Using interactive rebase lets you keep a clean and bisectable history made of small commits. The developer every now and then can also rebase to master and ensure that all commits pass the tests (and perhaps write more tests based on what happened in the meanwhile on the master branch).
However, when CI runs, features are included in master with a merge commit, so that the occasional semantic merge conflict will bisect exactly to the merge commit and the developer of the feature isn't blamed incorrectly.
> when one reads a real book about real world history, does it look more like git log history of a repo which has been using rebase, or merge?
This is a bizarre analogy. History books are a record of things that happened in the world's timeline, which is in fact linear. Source control is a record of things which happened in the timeline of development, which is typically branched.
Imagine a history book from some terrifying PKD-esque sci-fi universe where timelines branch and re-converge. Does that look more like git log history of a repo which has been using rebase, or merge?
Sure, I agree. But a history book tends to be more substantive than "WIP building a nation", which is what most of our commits look like in practice.
Squashing would seem to be the answer, but do you feel that's a bad idea? It certainly has tradeoffs. You can easily end up with a massive squash commit.
I really want to keep the WIP commits. They provide context even if their log messages don't, and they make git bisect easier. But I don't think anybody does that, and I'm curious why.
My bad, I was going to edit my comment to be less coy and more constructive, but I'll just continue here.
Even though I always advocate rebasing, I also think that WIP commits should not reach master as-is. We are actually using Phabricator with my current team, and they have a really nice opinionated way of doing development. All dev happens usually in (really short-lived) task branches, but once they have gone through review etc. they are landed on master as one single commit, with the commit message holding all the relevant information about the change(s) made in the original task branch.
The reason for not having the WIP commits on master is that "commit early, commit often" is good practice, so the sheer amount of WIP commits will completely drown out the actual, finalised changes (i.e. the "actually interesting history as in a history book") in master. So no, I don't think squashing is a bad idea, I think is absolutely essential if you rebase onto master. If you don't squash, rebasing might lead to more mess than using merge.
Now, as to keeping WIP commits. I think that their value is usually much overestimated. I can count the times I have needed to go back to the actual, raw WIP commits instead of the properly rewritten one in master with one hand. But if you feel that it's the only thing keeping you from switching from merge-intensive flow to a always-interactive-rebase one, I'd encourage you just to retain the original, short-lived development branches in origin as separate branches. IMHO that gives best of the both worlds, if you think throwing out the WIP commits could hurt too much.
---
Edit: oh, and having massive squashed commits should not usually be a problem because mostly they should not happen. Individual tasks should be so small that implementing them can not result in a massive amount of changed lines.
So, all dev happens in feature branches, and they're integrated into master as squash commits. That just leaves two questions:
- Use release branches? Or just take the shotgun approach of "everything in master has to be working all the time"?
- When a problem inevitably pops up and you have to roll back, how do you kill just one commit? It's already been pushed to the repo, so a hard reset wouldn't be a good idea, right? So I guess that points to using release branches.
It would be optimal if the team can have the discipline to never commit broken stuff, and we could live with "everything in master must be working all the time".
If that does not work in reality, or breaking it would simply cost too much (because mistakes will happen), I'd use release branch or use git tag to mark versions with "no really, this one really truly actually works in every way".
For your second point, see 'git revert'! It does exactly that, i.e. picks a commit and effectively removes it from a branch by making its "mirror commit". The other way is, of course, just deploying an actual fix asap, git revert is just for when fixing the problem is for some reason or other slow and master must be unbroken immediately.
Or most importantly for us, master needs to always work, because that's where you start new development on.
You can't expect to fix another bug somewhere else in the codebase if you are starting from a point where the code may not work, or features may only be partially complete.
I find the idea that master is the default is exactly why it should not be the development branch. For open-source software, that's what people download and try to build -- it should always strive to be production ready.
People should be downloading actual releases. This could be release archives, or release tags.
Version control is not primarily for consumption of releases, it's for participation in development, and I think the branching strategy should reflect that. Having the current development state be the default branch is entirely suitable for this purpose.
I've always found the use of "develop" vs "master" quite jarring, and not even that helpful for an end user. master can change on a whim, while using actual release tags gives you something stable.
> People should be downloading actual releases. This could be release archives, or release tags.
Yes, and those versions are represented by tags created on the master branch. Github automatically interprets tags created on the master branch as being releases, and even creates nice little tar files with the branch's contents.
> master can change on a whim
Actually, it can't. According to the Git workflow, the only thing committed to the master branch is either the contents of release branches or hotfixes.
You can have both. With the right tooling you can prevent direct push to master, but at the same time implement CI to which you can submit your branches to be merged to master.
This way you can develop against master, you won't push garbage into it by mistake, and you can guarantee that each (first-parent path) commit on master passes the automated tests.
Yes, because master is the default implicit branch. (Now, in git is possible to designate other branch as default, but folks who follow the advice in article often are not experienced enough to do it.)
I know a team that runs this branching model. It's quite surprising to hear statements "we never commit to master branch". "after cloning, always remember to switch to development branch". "when creating pull request, always remember if you used "development" as base". and then the mistakes eat up lot of cycles in the end.
I don’t think this article gives advice to not use master for development but rather what it means is:
1. master must only have stable, tested, and production (not production “ready”) code and nothing else
2. If something in production breaks the hotfix should go into master and strive to achieve step 1
In fact this is really a good model. Keeps development streamlined and disciplined.
This has nothing to do with breaking the idea of having sane git defaults. In fact it’s a sane utilisation of defaults.
Non default branching is a necessity anyway. Default branch usually means just a master branch and committing directly into master, I believe, would lead to disaster. If one has to create branches it should be done for lifestyle steps below production.
It is good exercise to actually learn this branching model and then use it in practice. You will soon realize that most projects will suddenly start taking unnecessary toll on you just because now you want to maintain multiple branches and it is a huge PITA.
Instead just follow this simple routine - stay as close to the master as possible. If a temporary diversion is needed, create a new branch (and maintain both master and the diversion for a while). Delete the diversion once the job is done.
If you are never able to delete the diversion, it is not your git branching model that failed, it is you and your code who failed. Diversions may be long term (and I hope you have the workforce to maintain that branch as well) but still finite time. How to prepare that diversion is a software engineering problem and not git's fault.
Yea I find this "successful" branching model in OP to be a huge pain. It does sound nice in theory but it's annoying in practice and honestly I see absolutely no benefit in adopting it.
This is how my team handles merge conflicts: your branch must cleanly merge for the pull request to be approved (which means you must resolve conflicts in your branch first). If there are two branches that may conflict in terms of functionality (but not at a source level) we call that out, and have the people involved reviewing both. When it comes time to merge, we usually merge the first one done to master, merge master to the second, then do extra testing in the second branch before merging it.
Sometimes if we know one branch blocks the other, we simply merge one to the other, but then still go to master separately. This makes the pull request review much simpler and keeps the code isolated while not duplicating effort. This works best when a big bug has a quick and simple but incomplete fix, and a more risky and complex but complete fix, or when there's multiple aspects to a new feature. We can decide to ship the first branch earlier if necessary.
That is a standard problem you face in all projects and the conflict resolution generally happens on a First Come First Serve basis. Of course, the maintainer could decide it on the basis of priority and then the person merging to master next is responsible to fix the new conflicts.
Do not use Git Flow for a web application deployed on your own infrastructure (SaaS, microservice, mobile backend, etc.). It will slow down development and make your software less reliable.
The entire purpose of Git Flow is saving up changes to release later, e.g., saving up for a weekly release event. Don't do that! Deploy your changes as soon as they are ready, if they aren't ready don't merge them into a shared branch. If you do Continues Delivery you don't need "hotfix" branches because every changes goes out as soon as it is ready, so you don't need any of the complexity of Git Flow.
By saving up changes for a release event it means more things are getting released at once. If there is a problem after deployment it will be harder to narrow down the cause. Git Flow fosters a harmful development mentality where developers merge untested changes to the develop branch, then move on, and expect someone to test and stabilize their changes before release. With trunk-based development (https://trunkbaseddevelopment.com/) or GitHub Flow (https://guides.github.com/introduction/flow/) developers take ownership of their code, and only merge to master after they have tested it. With a good deployment pipeline they can own their code all the way to production.
Git Flow also encourages humans to think about and make up version numbers, like 15.0.5. This is a pointless waste of brain power, web apps don't need version numbers. The artifact systems (packages, containers, etc.) may need something, but it can just be an incrementing number that no on thinks about.
Git Flow wastes so much time, and makes everything it touches so complex, all to enable the harmful behavior of saving up changes for later, and enabling the pointless use of version numbers.
Trunk-based development and Continues Delivery is the default way people develop, it is how you would work if you had a one person company with one customer. It also is how the biggest web companies in the world work. It scales from smallest to largest. Just use trunk-based development. Stay away from Git Flow.
> Git Flow also encourages humans to think about and make up version numbers, like 15.0.5. This is a pointless waste of brain power, web apps don't need version numbers.
Version numbers are used to represent specific states of the project in order to have fixed testable and auditable versions. It's what the user sees when he needs to check which software version he's using when talking about the software he's using, and what programmers need to know when they need to work on bugs/features present in previous versions of the software but not on others. If your app needs to be debugged and there are peopleother than yourself using, testing or working on the software, it needs version numbers. Otherwise, everyone will needlessly waste their time.
Version numbers waste as much brain power as knowing the name of someone you need to contact on a daily basis. You don't need to make up version numbers because plenty of people already did that. For instance, Semantic Versioning is a thing.
Notice I said "web application deployed on your own infrastructure".
While you said "[versions are] what the user sees when he needs to check which software version he's using when talking about the software he's using, and what programmers need to know when they need to work on bugs/features present in previous versions of the software but not on others."
In good SaaS products users don't see version numbers. What version of Gmail are you using? What version of GitHub? They don't have versions. There are also only two versions of your software, what is deployed and what us being deployed (or multiple "being deployed" if you have multiple canaries). Engineers can look up build numbers and changes in the CI system or other internal tracking tools without humans making up version numbers.
Semver is great for libraries and shipping applications, where third parties need to know about breaking changes so they can adjust their applications or configuration.
When deploying your own app to your own infrastructure, Semver wastes people's time. I've worked at a company that switched to Git Flow right before I started. I've seen so much time wasted by people discussing versions and branches. Should I merge to develop or a release branch, or is it a hotfix, should it be 15.1.0 or 15.0.1, or no, someone else claimed 15.0.1 before me, now I'll rename my branch to hotfix-15.0.2, oh 15.0.3 is done being tested before my 15.0.2, so they will rename to 15.0.2 and I'll rename to 15.0.3, oops I forgot to merge their changes into my 15.0.3 so I reverted 15.0.2, now I need to deploy 15.0.4 that has the changes from 15.0.2 and 15.0.3.
Seriously, no joke, this is what happens what you try to use Git Flow for a web app.
It's not awful, but there is room simplify it and tune it for your own needs. Using master as the develop branch is a fairly simple and sensible one. Most simpler workflows are basically git-flow trimmed down in some way. In a big team, I do like to keep release branches to isolate the polishing of the release from other development work. (Without it, teams tend to have a code freeze, which is unnecessary with a release branch.)
Because it's complicated and most people don't need that complexity at all. For some reason a lot of people happily jumped on the band wagon when this was released and started writing scripts to make it more bearable.
If you have multiple versions of your codebase that you need to maintain for a longer time to justify those release branches, gitflow might be for you. IMO GitLab flow is much more applicable for most people: https://about.gitlab.com/2014/09/29/gitlab-flow/
Because it creates a large amount of needless busywork.
It's much easier to develop on master and/or on feature branches that can be cleanly merged in to master, and then tag master (or create branches) once a release is hit.
I've used the git-flow approach successfully with a small team working on a medical product (so, embedded software system) -- every feature branch had to be reviewed before being merged with `develop`, which was submitted to nightly, extensive functional tests (initially one-hour long, eventually kept as a nightly subset of the more than 24-hours complete QA run) before it could be approved as a new (monthly) release and be merged with `master`. Every new feature branch was automatically treated to quick continuous integration tests, and available for manually-triggered full functional tests (on the target devices).
This approach ensured that we had a full trace of development work, (signed) code reviews, and software changes -- compatible with FDA audits.
We also automated collection of code coverage data during functional tests, to inform analysis and revisions of the battery of functional tests.
We used PRs with BitBucket for all code reviews. The reviewer(s) had to digitally sign their final approval of the review comments+answers and of the related code changes, if any.
The only way to merge a feature branch into `develop` was via the PR + code review process.
Was it something like exporting PR history to a file and then signing (X.509/PGP)? Thanks for answers, it looks like a nice, lightweight auditable system.
No, simpler than that; we used the BitBucket web interface to enter the approval message and click the approved button to allow for merge. These actions are recorded and visible in the overview page of the PR.
However the BitBucket server's web interface was not approved/validated for long-term storage and evidence for the audit trail, so the PR owner was responsible (before triggering the merge) for saving a PDF copy of that PR page and committing the PDF file into a git-controlled code-review directory.
So it's digitally signed to the extent that your account/identity is recorded in the approval step and in the collection of PDFs. I did ask about a more systematic export method but it was not considered important given the PDF-based approach.
This was very much valid before the docker workflows came to happen.
Now maintaining two mainline branches forces you to break "don't build a container per environment" cardinal rule. Trunk based development should be the go-to repository strategy for dockerized apps.
That's a good rule, and a correct deduction from it. However, it's not new to Docker workflows - you shouldn't build a good old fashioned binary per environment either!
If you properly externalized the configurations, Docker provides bit-by-bit parity between environments granted you use the same image.
This results in increased confidence to test environments and lessening the chances of a surprise during production deployments. It is the next logical step in immutable deployment paradigm[1].
But why not use the typical solution of having two testing environments, in this case based on "develop" and "master", with the latter being bit-by-bit equal to what gets deployed?
What purpose does it serve but to double QA efforts. If you are testing the same stuff both in develop and master, why not just test only one and save some time. Ramming gitflow into a docker workflow efficiently is not really possible I believe.
The chaos of huge feature merges (a key source of bugs I've experienced) is minimized. You deploy fixes hourly, not weekly (or later monthly when it just won't seem to pass CI). The time between code being written and a bug being seen can be reduced to minutes and hours, making finding the root cause a breeze.
Just my preference, but very open to debate.