“Is Feature Branching Evil?” by Jay Dorman, Director of Technology at Edgecast

Jay is a technology executive who runs application development and operations for a large team at Edgecast, a widely used and fast growing content delivery network based in LA. With broad responsibilities ranging from strategic decision making to hands-on management, Jay provides technical leadership and direction for his team. Jay shares his perspective on how and why he implemented Feature Branching at Edgecast.

Is Feature Branching Evil? by Jay Dorman

Remember when deploying a web application meant mapping a network drive and xcopying (or even dragging and dropping!) files between your development system and a production server?  And then dealing with all the files that didn’t copy because your webserver had them locked?  Come on…you can admit it…you’ve been there.  While my teams have moved FAR beyond those days, we’re still struggling with some aspects of deploying new features. We generally operate in two week sprints, which are followed by about two weeks of testing, break-fix and deployments before a release hits production.  That means there’s a nearly four week window between the time a feature bubbles to the top of the backlog and it sees the light of day…a pretty painful delay in anybody’s book.  It also means that some dozens of features are bundled together for release.  In the case where a feature needs to be delayed because of external dependencies, or because of a last minute bug, it forces us to roll back the entire release.  Yes, we can exclude the particular feature, but with a mature base of shared objects and base functionality, the altered deployment really needs to roll through QA and Staging again before it’s safe to deploy.  Even with a relatively high degree of automation (we use a combination of Jenkins, Octopus Deploy and RedGate’s tools), rollbacks and re-deployments can be pretty painful. We’ve decided to move towards feature branching.  If you ask The Google, it won’t take long before you’ll read/hear the phrase “…feature branching is evil.”  You’ll also read articles that describe feature branching as being in direct opposition to continuous integration.  I just don’t see it.  Lots of talking heads will tell you that in order to do continuous integration, you need to build logic gates around each feature so that development code can be disabled until it’s ready, then flipped on once it’s passed through all the quality gates, after which you’d remove those logic gates.  (Yes, I know…that’s a pretty simplistic explanation, but I think accurate for the purposes of this discussion).  This makes me nervous for a couple reasons.  First, I don’t like the idea of all these logic gates sprinkled throughout the code.  With a team of say, 10 developers writing features with broad scope, it seems like a lot of ‘meta code’ to write, test, remove and then re-test.  Second, what if you need to make changes to a broadly used core business object?  Wrapping conditional gates around the datatype of a property that’s accessed by dozens of methods and pages seems like a lot of risk. So, we’re moving towards a deployment methodology that we feel incorporates the best of feature branching and continuous integration.  Here’s some of the key points of the direction we’re heading:

• We’ve migrated from svn to git, as it’s a much better fit for the way we want to branch and merge changes.
• Each feature gets its own branch off of the main develop branch.  Incidentally, we’re naming each feature branch after the Jira ticket that tracks the effort.  Seems intuitive to us, and helps with automation.
• Throughout the development cycle, the developers merge up from the main branch into the feature branch.  This insures that features play nice with other changes that have already made their way into production.
• On a daily basis, we use Jenkins to compare each feature branch to main in order to make sure that developers are merging up every day.  With the linear nature of git, it’s easy enough to determine if a given branch is behind.
• Once a feature is code complete, we build to QA from the feature branch, before we merge to main.  That allows us to deploy urgent hotfixes and such, without inadvertently deploying untested features.  Also, when Jenkins initiates a QA build, it also spins up an OpenStack VM environment to test the feature.  That way we can have multiple changes going through QA independently and simultaneously.
• Once QA certifies a feature as ready for production, we deploy the same package to integration, staging and eventually production.  When that’s complete, we finally merge into main.
• Hard not to emphasize this again…with each deployment, we compare the build to what’s currently in main, merging up to the feature branch if necessary.

It’s taken a lot of automation, and loads of adjustment and practice, but we’re getting there.  Each feature goes through CI within its own branch, and if there’s problems, that’s where we catch them.  The developer(s) working in that branch have to address any integration issues before moving their feature towards production.  Every night we run automated unit tests and functional tests on each feature branch, and in a performance environment, we run performance, load and analytics tests.  Now features go out the door as soon as they’re ready.  Product feels empowered to prioritize and actually impact deployment timelines.  QA is happy because they don’t necessarily have to run full regression with each change (depending on the scope).  Execs don’t wonder “Why does it take so long to develop features.”  Maybe best of all, customers feel like we’re listening and quickly acting on their needs. Is it perfect?  I don’t know.  We’ll figure that out over time.  But we’re feature branching.  AND we’re doing CI.  And so far, we don’t feel evil.