Your Automation is Making You Less Agile

Most agile practitioners will tell you that you need to have automation. We need those rapidly running safety checks in place so that we can move quickly and not be slowed down by testing. I think few of us can imagine doing agile software development without a solid automated test suite to back us up. For most teams it is a given that we need this in place. This may be largely true, but the problem with givens is that we can stop thinking critically about them. Automation may be helpful, but is your automation being designed to make your team more agile, or is it just there to help you move more quickly?

You see, I think that is where we often get confused. We confuse being agile with being fast, but they are not the same things. Being agile is about being able to change directions quickly. Being fast is about how quickly you can move in the direction you are going. Now don't get me wrong, I think these are both important factors to effective software development, but do we really consider both of these factors when designing automation efforts? I suspect that much of our test automation is built to help us be fast and isn't built to help us be agile. Let's think through a few indicators that your automation might be making you less agile.

Mr. Glass

Remember Mr. Glass from the Shyamalan movie, Unbreakable ? The slightest fall or injury would break his bones. No matter how careful he was, he was constantly breaking something and hence the nickname Mr. Glass. Now think about your automation. Do you have any tests like this? Tests the break at the slightest breeze in the code. Maybe they are UI tests that break whenever UI elements change, or perhaps they are integration tests that fail for a wide range of underlying code changes, or they might even be unit tests that break any time you make a tweak to your functionality. Whatever the case, the fragility of these tests comes from assuming that the current answer will always be the correct answer. But if we are thinking about being agile we can't assume that. If agile really does mean the ability to change course, we need to think about that when writing tests. If my tests don't assume that things will change in the future, they are going to create additional friction when we try to change the direction we are going in.

Now, we do need a few caveats here. The ability to change course needs to be limited in some sense. For example no matter how agile you are trying to be, the ability to change your product from a biomedical product to a first person shooter video game would be ludicrous idea. There are things core to your work that you should expect to never (or very rarely) change. By all means write breakable tests for these. These are areas of the code where you want your tests to break at the slightest provocation, but there are many other areas that you should not assume will stay constant. In those areas of the code don’t write your tests with that assumption. Writing tests that assume things will not change is writing tests with the implicit assumption that the current plan is 'right' and that we will not need to respond to change. Believing we have it right from the start is not an agile approach to software development.

Wolverine

Wolverine doesn't need much introduction. He's a hard-bitten X-Men mutant with seemingly limitless regenerative powers. He just won't die. This is the second indicator of non-agile automation that I want to think about: the test that will not die. Part of being agile is realizing that you have things in the code that no longer need to be there and ought to be deleted. But what happens to the tests when we take some code out behind the barn and put a bullet in it? When we do that we should be able to quickly and easily delete the tests that are using that code as well, but is that the case? Far too often we have tests that can't be deleted because they are checking a bunch of other stuff as well. The tests might fail due to the deleted code (great!) but we can't just blow the test out of the water without concerns about missing other important checks. We end up having to do surgery on the test and removing the offending bits, or worse yet, we end up leaving the dead code in so that the tests will continue to work.

In simpler terms, Wolverine tests are those tests that are not doing one thing. A test that tries to do too much can't be easily removed and will end up making it harder for your team to change their mind or approach on something. I think that being able to easily delete code is an important factor to helping a team stay agile (For further research see here or here ) and of course that also means that your tests need to be easy to delete as well. Don't assume that because it make sense to have a test now, it will always make sense to have it. Think about how easy it will be to delete that test at some point in the future. Wolverine might be the kind of guy you want on your team if you're part of the X-men, but he isn't what you want when you are writing tests.

Spiderman

Having a secret identity is a such a common theme with superheroes that I had a hard time picking one, but let's go with Spiderman. If you know him as Peter Parker, you might think you know him well, but then one day you find out something and you are in for a big surprise. How many Spiderman tests do you have in your system? Tests that you think you know. You can look at the scripts and see a set of commands and everything seems ok, but then one day you find out that there is some dependency in there that reveals something else entirely about your test. An example of tests like this might be tests that rely on a database or a binary file load before they start. Everything works fine and you think you are just testing what is in the test's script, until the day there is a change to the expected structure of the underlying database or file, and your test breaks.

The problem with secret identity tests isn't that they have secrets as such, it is just that you don't really know or understand them and because of this they are a lot of work to fix and failures are hard to predict. Imagine that you have a suite of tests that load a database and then check different actions. If you change that database, it is very hard to predict which of the tests will fail in which ways. The use of that database can be somewhat opaque to you and may not be obvious from the test scripts themselves. This means that instead of a simple 'find-replace,' you need to run the test suite and figure out which tests are failing and why and see if the failures are indeed because of your expected database change, and then modify the tests to respond appropriately. A simple action has become a multistep headache, all because you let your tests have secrets. Expose those secrets!

Now What?

Now, nobody likes someone who is hating on superheroes - or on tests for that matter - so what can we do about this? Do we give up and just not automate in areas of the code that we want to have the flexibility to change in the future? Or is there some way we can change our approach so that we can leverage automation to allow us to be agile? I like automation and think it can be used much more than we already do, so I'm going to look at some of the things we can do to make our automation enable agile behaviour.

The first and most obvious thing is to think about it upfront. When you are writing an automated test (or working on an automation framework for that matter), ask yourself what will happen when this test becomes outdated or fails. Will it be easy to update, change or remove? Anticipate that the code will change and that your test will need to change as well and design your tests accordingly. The other thing that needs to be considered is your testing framework. In my experience most testing frameworks lead you in a certain direction and often that direction is not towards writing agile enabling tests. Being aware of this and accounting for it can help, but take it a step further as well. If you can add or advocate for new approaches and features in your framework, do it! Ask for the ability to do property based testing or the ability to track and expose dependencies. Ask for the ability to easily and automatically update tests. Ask for the ability to trace the coverage of your tests so you know what code is running as a part of each test. Look for tools that can do these things or help you do these thing and use them! Much of what we do is shaped by the tools we use so be aware of that and look for or create tools that will shape you in the right directions.

Automate away, but don't forget that the goal isn't to have as many tests as possible. The goal is to quickly produce valuable software. To meet that goal you will need to be able to shift your direction as the definition of what is currently valuable shifts. Write your tests to help meet that goal.