Your team is about to roll out the new AI-powered search feature the board has been pushing for. They put it behind a feature flag and selected a subset of users to take it live in production. You made sure that it was tested thoroughly and the team felt confident in releasing it. The team even made sure to test when the flag was switched on and off in a production-like environment. You did your due diligence and couldn't wait to message those users to begin testing after the release was completed. The release finishes, messages are sent, and you head home for the day.
The next day you are excited and can't wait for those first messages to arrive about your feature. You expect positive reviews, but you have good relationships with these users and expect them to tell you what's not working as well. Worst-case scenario, they don't like it, you turn the feature flag off and get back to the drawing board. Either way, you've been here before. Whatever comes back from them, you're ready.
When you arrive at the office, you are met with a slew of messages, but they aren't from the user pool testing the new search feature. They are from your engineering team. There was a partial outage in the system. Data loss and corruption followed. It is already affecting the search feature. The updates seem to be affecting a couple of services that are unrelated to the one that was updated during the release. Seems odd. You had your team do thorough testing before rolling it out. The first order of business is to restore the system, so you turn the feature flag off. Then the team starts working on restoring databases to stable states. It takes most of the day, but you restore the system. Clearly, there is a problem with the code behind the feature flag.
Over the next week, your team tests it again. No one can reproduce the issues. Code review comes up clean. You add some extra safeguards around the database interactions to try and avoid corruption if it were to happen again. The consensus is that it is safe to roll this out again. The updates are deployed to production and the flag is turned on. But once more, the same issue occurs. The same services are affected, and the same data corruption occurs. The good news is that this time the team can fix it much faster, so the system is restored within a couple of hours. No one understands why this is happening. And every day spent investigating is another day without that new feature that customers were promised. You don't know what to do.
If you've been here before, you feel this in the pit of your stomach. I've been here too. And not knowing why things are breaking hurts, especially when you've spent so much effort verifying the build. That's the key point: you only verified one version of the build. When you have feature flags, there are technically different versions of the system all within the same build.
The problem wasn't your testing of the feature. It was that you didn't test all the different code paths that are made through the use of the feature flags. When multiple teams are developing different features independently, it's easy to forget to test all combinations of feature flags. In fact, the team might not even want to raise this concern because it can exponentially increase testing demand.
Feature flags have a binary state, either off or on. That means the number of code paths to test is related to the number of feature flags (N) through this simple equation:
paths = 2N
For a single flag, you have 2 paths. For five flags, you have 32 paths. For ten, you have 1024 paths to test. Do you see how this can be a challenge to manage?
One single feature flag at a time is manageable. It can allow you to roll out that half-baked feature sooner and get the feedback. It makes you feel safer. But if you have to manage multiple flags, it doesn't make you any safer. You don't have the testing resources to guarantee reliability for every combination of off-on switches. Plus there is already inherent risk in releasing features under the flag.
If you were confident in your feature, you wouldn't be releasing it under a feature flag in the first place.
Shipping faster doesn't eliminate the need for thorough testing. If anything, it increases it. Feature flags don't make shipping safer unless you pair them with the same rigor you'd apply to any other release. They tempt you to skip the regression testing that you know you should run.
It's not all doom and gloom though. Feature flags are a tool, not a solution you can throw at every feature request. Use them sparingly and remove them as soon as the feature is stable.
The real question is harder: why didn't we just finish the feature in the first place? The speed of delivery isn't always worth the risk.