“With great test comes great… flakiness”

25-minute Talk

In our organization, we tackled test automation flakiness head-on by implementing a rigorous process to identify and fix flakiness, while shielding our testers from its effect on their daily work

Virtual Pass session

Timetable

10:45 a.m. – 11:30 a.m. Wednesday 15th

Room

Room E2+E3 - Track 5: Test Automation Deep Dive

Audience

testers, test automation engineers, production chain managers, infra-administrators,

Key-Learning

  • How to leverage historical data to build best strategies to reduce flakiness
  • How to use smart and automatic processes to identify the defects worth fighting for
  • How to build consensus and commitment within your organization to address this challenge
  • How to use gamification to add a fun factor to fixing flakiness

Sharing my experience and the experience of my organization with treating flakiness at-scale and in an industrialized fashion.

Before attending this version of the ATD conference, you probably have read numerous articles and watched numerous talks discussing test flakiness, the reasons behind it, and the ways to reduce it.

For this talk I would like to share my experience and the experience of my organization with treating flakiness at-scale and in an industrialized fashion. Some backstory: Our software factory has been producing and continuously evolving a single software product for more than 35 years, growing to be one of the biggest market leaders in the fintech industry, trusted by major banks and institutions to keep their business running and their numbers straight. This could not have happened without a strong attention to quality and huge investment in testing.

I and other QA consultants in my organization, we pride ourselves with the testing culture that we have built over the years and with the test repository we have developed, reaching today an arsenal of more than 1200 online tests that fully deploy and test our software, with a frequency of 20,000 instances of tests executed daily amounting to 20,000 hours of test runtime.

So, you can imagine on this scale how flakiness becomes a fact we cannot ignore, the impact of which can disrupt our production chain and frustrate our teams. A solution has been put in place by the QA department and is being continuously enhanced to tackle this problem, and I am eager to share with you the steps we took to reach our current model:

• How we introduced short term patches while investing in long term resolutions

• How we leveraged historical data to build our strategies

• How we used smart and automatic processes to identify the battles worth fighting for

• How we gathered consensus and commitment within the organization to address this challenge

• How we used gamification and mock awards as a team building activities adding a fun factor to our recipe

• How our model can be further enhanced to cover more aspects of test health

• And finally, how it can be tailored to fit your organization

Related Sessions

Virtual Pass session
9:15 a.m. – 10:15 a.m.
Room F1+F2+F3 - Plenary

45-minute Keynote

8:30 a.m. – 4:30 p.m.
F-,E- & D-Rooms

Full-Day Tutorial (6 hours)

Virtual Pass session
10:45 a.m. – 11:30 a.m.
Room F1 - Track 1: Talks

25-minute Talk

8:30 a.m. – 4:30 p.m.
F-,E- & D-Rooms

Full-Day Tutorial (6 hours)