Since Timothy first wrote about it, IMVU’s bleeding edge continuous integration strategy got a lot of attention. Some of it has been negative, and some of that negative attention has been well-researched. I want to address one part of it in particular. There appears to be a feeling in the mind of Michael Bolton, or at least of the comments on his blog, that continuous integration causes bugs and general low quality. This is completely wrong, and I feel like it needs addressing.
I’m a software engineer at IMVU, and as such first-hand consumer of our continuous integration practice.
The nature of this apparently causal link between continuous integration and quality has been speculated upon. Rather than address any wrong theory, I’ll just tell you what’s really going on. Continuous integration doesn’t make us write buggy code, it lets us get away with writing buggy code.
To see why, you first have to recognize bugs have two costs. They make your product harder to use, and they make it harder to maintain. You suffer the first by losing customers, and you suffer the second through the payback of the technical debt. You have to fix bugs as early as possible so they don’t infect your new software.
What TDD and continuous integration have done is free us from the second cost, allowing us to defer the payback on the first. Bugs never cause significant regression in new software, because our dedication to test-first development keeps those bugs form propagating into new features. This is not to say that we never fix bugs; it means that there’s a lower ROI on fixing them. We can choose to delight customers by making new things for them instead of by fixing the old things, and only fix the bugs that have significant customer impact. Nobody cares if some page nobody uses doesn’t work right.
You might think that this would cause our customers to hate us. While it makes some customers angry, it doesn’t increase the overall level of dissatisfaction. Different people get angry about different things. At any given time, some of our customers are complaining in our forums, which, unlike Apple, we do not censor for complaints. But, you can’t please everyone; when we fix bugs, customers complain that we’re not adding new features.
Since our tests prevent our bugs from hampering our development process, we have the choice to ignore them indefinitely. It then becomes a product management level decision whether to fix a given bug, rather than a required engineering-level task. You may think we’re making the wrong decision, but understand that you’re disagreeing with product level decisions, not with the decision to use continuous integration. Those product level decisions might seem stupid to you, but those guys are using some pretty sophisticated data-driven decision making, and it’s working.
Integrable Is A Word 