(Almost) zero regression bugs
Henri Karhatsu (@karhatsu)
Case project at Itella:
Muistajan osoitekirja / Well-wisher's Address Book
The first version was deployed to production in Jan, 2012.
Nine months later there has been (almost) zero regression bugs.
Regression bug
=
"This thing has worked before but it
doesn't anymore after the latest update."
Reason 1. ATDD (acceptance test driven development)
Before implementing a feature write an acceptance test for it.
ATDD tool in this project: Robot Framework
User can login with correct username and password
Setup user my-username my-password
Go to the login page
Fill username my-username
Fill password my-password
Click login button
Verify that user has successfully logged in
...is translated to executable test code, which is run against the system.
Reason 2. TDD (test driven development)
Before writing any code write a failing test for it.
In a way TDD is very simple, on the other hand it's very hard.
It really helps when the whole team believes it and
there is no legacy (=untested) code.
Reason 3. Several feedback points in our (Kanban) process
Feedback points in the process:
a) Planning stage - Before a developer can start to implement a feature, it must have pending Robot tests that have been presented for another developer and accepted by the Product Owner.
Feedback points in the process:
b) Coding stage - TDD & pair programming.
Feedback points in the process:
c) Code review stage - Feedback from another developer.
Feedback points in the process:
d) Testing stage - Feedback from the tester.
Feedback points in the process:
e) Acceptance testing stage - Feedback from the Product Owner.
Feedback points in the process:
f) Jenkins - Feedback from continuous integration.
Reason 4. Good tester
It's difficult to write all required tests beforehand. That's why we need a good tester who is able to show that you forgot to write a test for this case.
On the other hand since we have a good test suite, the tester can concentrate on exploratory testing instead of repeating the same steps all over again.
Reason 5. Respect quality
As an example: In this project there was a limit for technical debt. The team was allowed to have a maximum of five recognized quality debt items.
Conclusion of five reasons:
Automate testing as much as possible. Be disciplined.
Let the tester use his expertise, do not make him a manual testing machine. Respect quality.
Simple things, no magic.
So what do you mean with "almost" zero regression bugs?
During nine months there has been zero functional regression bugs but a couple of ones related to layout. And mostly related to IE7...
Lesson related to "almost":
Even though everything cannot be automated, good test coverage gives the tester a better possibility to concentrate on the tricky cases. Like IE7...
How about bugs related to new features,
why do you talk about "regression" bugs only?
Unfortunately I do not have statistics about the bugs related to new features. Nevertheless, the amount is not very high.
Bugs in new features appear since we can't always realize what all we need to test. But what really matters is that how fast are you able to fix the bug after you notice that. And there it helps enourmously if you have a good test suite.
After bug has been found:
1. Write a test that fails and thus proves the existence of the bug.
2. Implement the bug fix and see the test passing.
3. Be happy with the even better test suite.
4. Never see regression in that part of the system.
So it was the greatest project ever. Nothing to improve, right?
How did we test that we were building the right thing?
During the project we did not.
