Step-by-step: Programming incrementally
One thing that has really benefited my productivity (and also my general sanity), has been learning how to take a big task and break it down into smaller, more manageable steps. Big tasks can be frightening and overwhelming, but if I just keep working on the list of smaller tasks, then somehow, as if by magic, the big task gets completed.
Adding value
When programming, I take a very specific approach to this breakdown. I make sure that each step is something that compiles, runs, passes all the tests, and adds value to the codebase.Exactly what “adds value” means is purposefully left vague, but it can be things like adding a small feature, fixing a bug, or taking a step towards refactoring the code into better shape (i.e., reducing technical debt).
An example of not adding value is adding a new feature, but also introducing ten new bugs. It’s not clear that the value of the feature outweighs the cost of the bugs, so it might be a net loss.
Another example of not adding value is making the UI prettier, but also make the app run ten times slower. Again, it’s not clear that the prettier look is worth the performance hit, so it might be a net loss.
Of course, I can’t always be sure that I’m not introducing bugs, and “value” is inherently subjective (how much performance is a new feature worth). The important part is the intention. My intention is to always add value with every single commit.
Basically, what I want to avoid is the “It has to get worse before it gets better”-attitude. Also known as: “The new system is the modern way to do it. Sure, it has some bugs and runs kind of slow right now, but once we’ve fixed that, it’s going to be way better than what we had before.” I’ve seen too many cases where these supposed fixes never happen and the new system, which was supposed to be better, just made things worse.
Plus, you know, it feels good to add value. If every day I can make a commit and that commit makes the engine better in some way, that makes me happy.
Pushing to master
In addition to implementing changes through a series of small commits, I also push every one of those small commits back to the master branch.
Note that this is the exact opposite of a feature branch workflow, instead, it is a form of trunk-based development:
In a feature branch workflow developers work on new features in isolated, separate branches of the code and don’t merge them back to master until they’re “complete”: fully working, debugged, documented, code reviewed, etc.
In the trunk-based approach, features are implemented as a series of small individual commits to the master branch itself. Care must be taken so that everything works even when the features are only “partially implemented”:
Trunk-based vs feature branch development.
Trunk-based vs feature branch development.
Proponents of the feature branch approach claim that it is a safer way to work since changes to the feature branch don’t disrupt the master and cause bugs. Personally, I think this safety is illusory. Bugs in the feature branch just get hidden until it’s merged back to master when we suddenly get all the bugs.
Feature branches also go against my philosophy that every commit should add value. The whole idea behind a feature branch is: “we’re going to break a bunch of shit over here, but don’t worry, we’ll fix it before we merge back to master”. Better not to break stuff in the first place.
Here are some other advantages I see with the trunk-based approach:
Fewer merge conflicts. With long-running feature branches, the code in the branch drifts further and further away from the code in master, causing more and more merge conflicts. Dealing with these is a lot of busy work for programmers and it also risks introducing bugs. Some of these bugs won’t be seen until the branch is merged.
Less release-day chaos. Typically, all features scheduled for a certain release have the same deadline. This leads to all feature branches being merged just before the deadline. This means that we get all the merge and integration bugs at the same time, just before the release date. Getting a lot of bugs at the same time is a lot worse than having them spread out evenly. And getting them just before a release is due is the absolute worst time to get them.
No worry about the right time to merge. Since everybody knows that merging a feature branch tends to cause instability, this leads to worry about the “right time” to merge. You want to avoid merging right before a release (unless the feature is required for the release) to avoid introducing bugs in the release. So maybe just after the release has been made? But what if we need a hotfix for the release? While the merge is being held, valuable programmer time is being wasted.
No rush to merge. When working with feature branches, I often felt a hurriedness about getting the branches merged. Sometimes because a branch was needed for a specific release. But also often because the developer was tired of dealing with merge conflicts, wanted to get it over with, and move on to the next thing. Thus, the goal of only merging feature branches when they are “complete” was often compromised. (And of course, nothing is ever really “complete”.)
Easier to revert. If major issues are discovered after the merge of a feature branch (which often happens), there is often a lot of reluctance to revert the merge. Another big feature branch might already have been merged on top of it (since lots of feature branches often get merged at the same time, just before a release), and reverting it would cause total merge chaos. So instead of doing a calm, sensible rollback, the team has to scramble desperately to fix the issues before the release. With trunk-based development, any major issues would most likely have already been discovered. The final commit that makes the new feature “go live” is typically a simple one-line change that is painless to revert.
Partial work is shared. In trunk-based development, the partial work done on a feature is seen by all developers (in the master branch). Thus, everybody has a good idea of where the engine is going. Bugs, design flaws, and other issues can be discovered early. And it is easier for others to adapt their code to work with the new feature. It’s also easier to get an estimate of how much work is needed to complete a feature when everybody can see how far it has progressed.
Easier to pause and pick up later. Sometimes, work on a feature might have to be paused for a variety of reasons. There might be more critical issues that need to be addressed. Or the main developer of the feature might get sick, or have vacation coming up. This is a problem for feature branches because they tend to “rot” over time, as the code base drifts further and further away, causing more and more merge conflicts with the branch. Code that is checked into master does not “rot” in the same way.
Easier to address other bugs/refactors at the same time. When working on a feature or a problem, it is pretty common to find other, related problems, exposed by the work you are doing. In the trunk-based approach, this is not an issue. You would just make one or more separate commits to the trunk to fix those issues. With the feature branch approach, it is more tricky. I guess the right thing to do would be to branch off a new separate bug fix branch from master, fix the issue in that branch, merge that branch into the branch you are currently working on, and (once it passes code review) into master (so that other people get the bug fix before your feature branch is merged, because who knows when that will happen). But who has time for all that shit? So instead, people just fix the problem in their feature branch, and maybe cherry-pick it into master if they’re having a good day. So now, instead of being about a single isolated feature, the feature branch becomes a tangled mix of different features, bug fixes, and refactors.
The main challenge of the trunk-based approach is how to break a big task down into individual pieces. Especially, with the requirement that each piece should compile, run, add value, and be ready to be pushed into master. How can we push partial work without exposing users to half-baked, not yet fully working features?
Let’s look at some problems and how to solve them.
Problem #1: New features
An approach that works well for new features is to use a flag to control whether a feature is visible to end-users or not.
Let’s look at an example. A feature that I recently added to the engine was a Download tab that lets the users download new engine versions and sample projects from within the engine itself:
Download tab.
Download tab.
There are lots of different ways this could be broken down into smaller steps. Here’s an example:
Add the Download tab to the menus and show a new blank tab when it’s opened.
Show a (hard-coded) list of things to download (without working download buttons).
Download the list of files from a server instead of having it hard-coded.
