Observability - From Fire Fighting to Smoke Detection

The Importance of Observability – From Fire Fighting to Smoke Detection

How the nightmare of losing weeks of data transformed our teams way of working

In the past we tried to test as much as possible up front and reacted with customer issues as they arose. Traditionally we didn’t have a focus on observability, now we have the ability to monitor and detect problems as soon as they occur in production.

With the benefits of continuous delivery and microservices comes the challenge of distributed systems. We can no longer just deploy into production and assume it works. We need to monitor in production and see how the customer uses our features.

In this story I’ll recount how our team went from a system of firefighting customer issues to a situation where we were able to detect issues in production before the customer was aware and resolve them quickly. How we implemented Monitoring, Logging and Alerting tools to help combat this. Now our whole team is focused on observability during design and planning instead of trying to retrofit monitoring tools as an afterthought once features have been deployed.