NOV. 3 – 8, 2019
POTSDAM, GERMANY

EUROPE'S GREATEST AGILE SOFTWARE TESTING FESTIVAL!

Hunting for Bad Data

A Practitioner’s Guide to Self Healing Systems

Defects in our data are as important as the defects in our code.

In 2013, the total amount of data in the world was 4.4 zettabytes. In 2020 it is estimated to be 10 times more. With speed advances and miniaturization of computers, it is now easier than even to collect all sorts of data in vast amounts. But quantity does not equate quality. Some of the most expensive software defects were caused by handing an incorrect data. The most recent example is the crash of the Schiaparelli Mars Lander.

Back to Earth, at Falcon.io, we investigate every exception that is generated from our production environment. We were surprised to find out that 19% are caused by bad data. This includes missing data, data with the wrong type, truncated data, duplicated data, inconsistent format etc. As our core business is to collect and process and analyze data from the biggest social networks, this finding was a major wakeup call.

“Jidoka” is a term that comes from lean manufacturing. It means to give the machines “the ability to detect when an abnormal condition has occurred”. We wanted to go one step further and create a simple, yet robust system ability to not only recognize bad data but also to fix it automatically so that it can heal itself. As it turns out, in some cases, it’s faster/easier to fix bad data than to locate the code that produced it. This talk is a practical one, containing ideas and tips that you can quickly implement when dealing with data quality.


More Related Sessions


30-minute Talk

14:25-14:55 Room F2 - Track 2: Talks

Ready Tester One? Go!

Equipment required

30-minute Talk

11:10-11:40 Room F2 - Track 2: Talks

Test Impact Analysis.

120-minute Workshop

10:25-12:25 Room D5+D6 - Track 8: Workshops

Negotiation: How to Master Critical Conversations

Full-Day Tutorial (6-hour Workshop)

9:00-17:00

A Testing Path to DevOps

Other Events:

Your privacy matters

We use cookies to understand how you use our site and to give you the best experience on our website. If you continue to use this site we will assume that you are happy with it and accept our use of cookies, Privacy Policy and Terms of Use.