Deep Oracles

One of the most widely touted drawback of the automated tests is that they work in strictly bounded context. They can only detect problems for which they are specifically programmed. The standard automated test has a bunch of assertions in the last step. The outcome of the test (pass/fail) is decided by those assertions. By definition, an automated test cannot detect an ‘unknown’ problem.

Because of their narrow focus, the automated test are occasionally compared to dumb robots. It takes a lot of time and effort to write and support them however their return of investment is still marginal. I’ve heard this mantra so many times that people just starting in the testing field can easily accept it as a truism.

Using the 6 techniques (flaky behavior, random test data, attack proxy, logs insights, data quality and application metrics), any automated tests can be transformed into sensitive and highly advanced suite. This suite will be able to detect problems for which the tests are not specifically programmed. New, unseen or unanticipated problems are now immediately highlighted. The value of your tests will be dramatically increased. And the best part is that you don’t need to modify the existing tests.