Picking a branching strategy has far-reaching consequences: it determines how developers collaborate, how code gets deployed, and how easy it is to understand changes when debugging.
For mobile app developers, life is rather cumbersome because continuous deployment is out of the question. One can still continuously deliver a release-ready binary, but the way App Store and Play Store work make it impossible to deploy every release to end-users.
Teams that work on fast-moving apps often create release branches from the main branch where builds are stabilised and tested. This also allows developers to continue working on the main branch and not get blocked by a code freeze.
Let’s look at the three most common patterns used by app teams.
Gitflow has been around for quite some time, and there are lots of references on how to implement it. It is considered a legacy pattern today because it has the propensity to create merge hell around integration time, and managing so many long-lived branches is a cognitive overhead for the team. However, because it was popular for so many years, it has continued to exist inside many large organisations and legacy codebases.
Most versions of it have:
Once a release is made, someone manually merges the dev branch into the main branch, usually creating a tag on the merge commit. This tag represents the state of production.
If a hotfix has to be made against the production build… pain ensues. We’ve interviewed more than thirty mobile engineering teams, and every team that is still relying on Gitflow, uses one of the following approaches:
The obvious issue with this approach is that it is needlessly complicated and must be done manually. No commonly available tool will manage this branching/hotfix complexity for mobile teams.
Merging long-lived branches is a big overhead of using Gitflow and one of the reasons why it slows down release cycles. After every major merge teams have to ensure that the state of code in the integrated branch represents what they want to release. Often for teams, it takes many days of manual work to test these release builds!
However, the biggest advantage of Gitflow is that developers don’t use feature flags for merging WIP work-in-progress code, so no technical debt is incurred by using flags themselves. Of course, they may still use remotely-controlled flags for dark rollouts or A/B testing.
Most people don’t realise this but trunk-based development (henceforth referred to as TBD) predates Gitflow. As the official website on TBD says:
"It has been a lesser-known branching model of choice since the mid-nineties and considered tactically since the eighties. The largest of development organisations, like Google (as mentioned) and Facebook practise it at scale."
The broad ideas behind TBD are:
TBD is a prerequisite for practising Continuous Delivery. If developers are not merging code to the main branch frequently, then code is not being continuously delivered. With patterns like Gitflow, there are often days or weeks of delay before code reaches production.
The issue with practising TBD is that no popular code hosting platform supports it out of the box. Phabricator supports trunk workflows, but it isn’t maintained anymore. Not all is lost though – we hope that new platforms like Trunk.io Merge will continue to emerge so that TBD is easy for everyone to practise.
Specific to mobile app development, TBD can be implemented in two ways: “Pure Trunk” and “Almost Trunk”. This nomenclature is our invention; these words are not used in the community, but we’re going to use them to describe the two methods.
In Pure Trunk, developers follow what the official TBD website describes. Many teams at eminent organisations like Google and Facebook follow this approach.
Everyone directly commits to the main branch, and developers do not create any branches on the remote server – they may create local branches for their own convenience though. The system ensures that all pushes to main are fast forwards, so developers will pull --rebase the main branch from the remote server before pushing their local changes. Code that is WIP is also continuously pushed to main but kept behind a feature flag.
Code review in such a system is done by holding commits in a special area before they are merged to the main branch – internal tooling does this automatically. The tooling will also continuously check for mergeability of commits as the main branch is always moving forward.
Releases are just tags created on main. If any issues are discovered when verifying a release, the fixes are simply committed directly to main, and a new tag is created for the fixed release. This also makes it ridiculously easy to create a hotfix for a production build – it’s the exact same process! The hotfix commit is created on main and a new tag is created for the release.
Almost Trunk is much easier to implement because it needs no special tooling, so it can be practised using Github or Gitlab. For example, the Simple.org Android app uses this approach.
Instead of directly committing to the main branch, everyone creates short-lived branches from the main branch for all work. These branches are merged into main within 24-48 hours, which requires some discipline from the team. If a feature is finished within that time frame, it is merged without a feature flag. Otherwise, the WIP code is put behind a flag and then merged.
When doing releases, a special release branch is created from the latest commit on the main branch. We say latest because, in the philosophy of trunk, all commits on the main branch are release ready and main is never broken. The purpose of the release branch is to hold any release-specific fixes that may need to be done as the release build is tested and verified.
However, there is a key difference between Gitflow and Almost Trunk: commits are never made directly on the release branch. All fix commits must be created on the main branch (using the aforementioned short-lived branch process), and then those fix commits are cherry-picked onto the release branch. This ensures that everyone else on the team who is continuing to deliver work gets the fix immediately. It also means that the release branch does not need to be merged back into main; it is simply a holding branch that can be ignored once the release has been completed.
Hotfixing a production build follows the same process: the hotfix commit lands on main and is then cherry-picked onto the relevant (usually the latest) release branch, and a new release is created.
Practising either form of TBD without having automated tests is very difficult because the main branch is moving forward all the time, and all commits must be release-ready. Teams that do this well have a mix of unit and integration tests that are run on speedy hardware, against every commit.
Without automated tests, if someone breaks main they will probably never realise what they’ve done. Every developer who delivers work after that broken commit is also (perhaps unknowingly) working on a broken branch. This will cause a lot of pain for the team while also making the main branch un-release-worthy.
Flags have always been a double-edged sword but implementing TBD without them is impossible. Whether working on features or fixes, developers may take many days or weeks to write code. The only way to keep WIP to a minimum is to keep merging that code but keep it behind a flag.
For app teams, it is helpful to have a special screen in internal builds which people can use to toggle features on or off. It allows non-technical stakeholders (think the product and design teams) to try out work that is in progress, without involving developers to provide builds with specific features turned on. An example of such a flags screen can be seen in the Google Chrome browser by going to chrome://flags.
This is not exactly a prerequisite for TBD but having easy access to intermediate builds from feature branches – even short-lived ones – makes life much easier for stakeholders to verify what is landing on the main branch. For example, if every short-lived branch generates an app build and sends it to a channel (Slack, Firebase App Distribution, internal S3 bucket etc.), anyone on the team can download that build and install it on their test device. They may not test every single build but the process is very easy when that need arises.
So what should your team pick?
If you’re a small team – say less than five developers – Gitflow should work fine. Coordination is not a big issue at that team size, and it’s probably not worth incurring the extra overhead of following TBD. You can simplify Gitflow by discarding the dev branch and having everyone work off the mainbranch directly. Feature branches, release branches, and tagging remain the same.
For larger teams, it is worth the time and effort to invest in Almost Trunk. As your product and teams grow, TBD will scale well and continue to give you the benefits of effective release engineering.
Many teams rely only on manual testing for their mobile apps, but even for those teams, we recommend the Almost Trunk pattern. Manual methods take so much time that your team can only verify a limited number of builds. Often, these end up being just the integration builds – commits which contain finished features, specific bug fixes, or the ones stabilising a release. Those integration builds get created in all branching strategies. Let’s see some examples:
You get to do everything you do in your current release process, but with a branching model that is far simpler to understand.
Microsoft’s CodePush framework enables React Native apps to deploy JavaScript bundles of their app over-the-air, which is as close to continuous deployment as one can get. CodePush allows developers to bypass sending updated binaries to the App or Play Store for every small change that has to be delivered to users. It is a promising piece of technology that is officially blessed by the App Store and Play Store policies.
However it comes with limitations: It only works with apps built using React Native and Cordova, and the success rate of update delivery is mediocre at best. In our conversations with React Native teams, CodePush reliability repeatedly came up as a problem. It is possible to verify this by looking at store binary update data for popular React Native apps because those apps still do frequent deploys to the stores even with access to CodePush.
___________
Thank you to Abhinav Sarkar, Apoorv Khatreja, Arnav Gupta, Kinshuk Sunil, Saket Narayan, Sanchita Agarwal for reviewing drafts of this post.
