Production Monitoring and Observability: From Blind to Insight

Jan 20, 2025

I used to run production systems blind. When something broke, I’d find out from angry users tweeting at us or from my phone buzzing at 3 AM with a vague alert that said “service unhealthy.” I’d spend 30 minutes just figuring out what was broken before I could even start fixing it. The mean time to detect was measured in hours. The mean time to understand was measured in frustration.

Over the past four years, I’ve built a comprehensive monitoring and observability system that detects issues before users notice them, tells me exactly what’s wrong within seconds, and has reduced our mean time to resolution from hours to minutes. This didn’t happen overnight. It was a gradual evolution driven by painful incidents and the realization that you can’t fix what you can’t see.

Let me walk you through the three pillars of production observability and how I learned to implement each one.

The Problem: Flying Blind in Production

Three years ago, we had what I generously call “monitoring.” We had a few CloudWatch alarms that would page us when CPU went above 80%. We had application logs scattered across dozens of servers with no centralization. We had no metrics about application performance, just infrastructure metrics. When users reported issues, we’d start SSH-ing into servers and grepping logs hoping to find clues.

The Incident That Changed Everything: One Friday afternoon, our checkout service started returning errors. Users couldn’t complete purchases. Revenue was dropping. I got paged. I SSH’d into a server and started tailing logs. Nothing obvious. I checked CPU and memory. Normal. I checked database connections. Fine. I spent 45 minutes investigating before I discovered that a third-party payment API we depended on was timing out.

The worst part? The payment API had been timing out for 2 hours before anyone noticed. We’d lost tens of thousands of dollars in revenue. And we only discovered it because users complained, not because our monitoring detected it.

I spent that weekend reading everything I could find about observability. By Monday, I had a plan.

Pillar 1: Metrics and Alerting with Prometheus

The Problem: We had no visibility into application-level metrics. We didn’t know request rates, error rates, latency percentiles, or anything about what our applications were actually doing. Infrastructure metrics (CPU, memory) weren’t enough because most application problems don’t manifest as high CPU.

The Solution: I implemented Prometheus for metrics collection and alerting. Prometheus became my single source of truth for everything happening in production. I instrumented every service to expose metrics (request counts, error rates, latency histograms, business metrics). I set up Prometheus to scrape those metrics every 15 seconds. I built dashboards in Grafana to visualize everything. I configured alerting rules based on actual user impact, not arbitrary thresholds.

The transformation was immediate. Within a week of deploying Prometheus, I detected three serious issues before users noticed them. A memory leak that would have caused an outage in 6 hours. An API endpoint with a 50% error rate that only affected a small subset of users. A database query that had suddenly gotten 10x slower.

What I Monitor Now:

For every service, I track the four golden signals (latency, traffic, errors, saturation). I have dashboards that show me the health of every service at a glance. I have alerts that fire when error rates spike, when latency crosses SLOs, when saturation approaches limits.

I also track business metrics in Prometheus. Completed checkouts per minute. User signups per hour. Payment processing success rate. These business metrics are often better indicators of problems than technical metrics. If checkout completions drop by 50%, something is very wrong even if all the technical metrics look fine.

Read the full guide: Prometheus Monitoring Best Practices

What you’ll learn:

Pillar 2: Log Aggregation and Analysis

The Problem: We had logs scattered across hundreds of containers. When debugging an issue, I’d have to kubectl logs into multiple pods, grep for relevant lines, try to correlate events across services. It was slow, manual, and error-prone. Complex issues that spanned multiple services were nearly impossible to debug.

The Solution: I implemented centralized log aggregation using the EFK stack (Elasticsearch, Fluentd, Kibana) initially, then later migrated to Loki for better cost and performance. Every log line from every container flows into a central system where I can search, filter, and analyze across all services simultaneously.

The key insight was structured logging. I enforced a standard JSON log format across all services with consistent fields (timestamp, level, service name, trace ID, message). This made logs machine-readable and searchable. The trace ID was particularly valuable because I could follow a single user request across multiple services.

A Real Debug Story: We had reports of intermittent checkout failures. Only about 2% of checkouts were failing, and the failures seemed random. With centralized logs and trace IDs, I could search for all failed checkout requests, find their trace IDs, then follow those traces across our frontend, API gateway, checkout service, payment service, and notification service.

What I discovered was that the notification service was occasionally timing out when trying to send order confirmation emails. When it timed out, it would throw an exception that bubbled up and caused the entire checkout to fail even though the order had already been processed. The user’s money was charged but they got an error message.

Without centralized logging and trace IDs, this would have taken days to debug. With proper log aggregation, it took 20 minutes.

Read the full guide: Log Aggregation Strategies

What you’ll learn:

Pillar 3: Proactive Infrastructure Testing

The Problem: Most monitoring is reactive. An alert fires after something breaks. Users are already impacted. I wanted proactive monitoring that would catch problems before they became outages. I wanted to test that our infrastructure actually worked, not just that servers were running.

The Solution: I implemented active infrastructure testing using a combination of synthetic monitoring and chaos engineering. Synthetic monitors constantly probe our production services pretending to be real users. If the synthetic monitors fail, we know there’s a problem before real users are affected. Chaos engineering (controlled failure injection) validates that our systems handle failures gracefully.

I set up synthetic monitors for all critical user journeys. One monitor signs up a test user, logs in, performs a search, and completes a checkout every 5 minutes. If any step fails, we get paged immediately. This has caught numerous issues during off-peak hours before real users encountered them.

For chaos engineering, I used Chaos Mesh to inject controlled failures in production. Kill random pods. Inject network latency. Fill up disks. The goal is to prove that our systems are resilient and that our monitoring detects failures correctly. Every chaos experiment has found something we needed to fix, usually gaps in our monitoring or assumptions about how systems would fail.

A Chaos Story: I ran a chaos experiment that killed all instances of our user service simultaneously. I expected the load balancer to stop sending traffic, the pods to restart automatically, and service to resume within 60 seconds. That’s not what happened.

What actually happened was the pods restarted, but the readiness probes failed because the service tried to warm up its cache from the database, and cache warming took 2 minutes. During those 2 minutes, no pods were ready, so all user service traffic failed. Our monitoring showed “pods are restarting” but didn’t alert because we assumed restarts were normal.

I fixed this in two ways. First, I improved the service’s startup time by making cache warming asynchronous. Second, I added an alert that fires if all pods in a deployment are not ready for more than 30 seconds. Without chaos testing, we never would have discovered this issue until a real incident.

Read the full guide: Infrastructure Testing Strategies

What you’ll learn:

The Complete Observability Stack

Here’s how all three pillars work together in production:

Metrics (Prometheus/Grafana):

Logs (Loki/Elasticsearch):

Proactive Testing:

The Workflow When Incidents Happen:

  1. Alert fires from Prometheus (high error rate detected)
  2. On-call engineer checks Grafana dashboard (which service is affected?)
  3. Engineer searches logs in Kibana/Loki using trace IDs
  4. Engineer follows the failing request across services
  5. Root cause identified within minutes
  6. Fix deployed, verified by metrics returning to normal

This workflow has reduced our mean time to resolution from hours to minutes. We went from flying blind to having full visibility.

What This Actually Achieved

The numbers from four years of building observability:

Detection:

Resolution:

Proactive Prevention:

Business Impact:

Lessons Learned From Building Observability

Start with metrics before logs. Metrics tell you when something is wrong. Logs tell you why. You need metrics first to detect issues, then logs to debug them. I initially focused on logs and ended up with lots of data but no way to know when to look at it.

Alert on user impact, not system state. An alert that fires when CPU is above 80% is useless if 80% CPU doesn’t affect users. An alert that fires when 95th percentile latency exceeds your SLO is useful because users are experiencing slow requests. Design alerts based on what users actually care about.

Trace IDs are essential for distributed systems. Without trace IDs, debugging issues that span multiple services is nearly impossible. The investment in propagating trace IDs through all services pays off the first time you need to debug a complex failure.

Structured logging is worth the effort. Making developers adopt a standard JSON log format was painful initially. But once we had structured logs, we could build automated analysis, search logs efficiently, and correlate events across services. The ROI was immediate.

Chaos engineering finds real issues. Every chaos experiment I’ve run has uncovered problems. Gaps in monitoring. Incorrect assumptions about failure modes. Race conditions that only appear under specific failure scenarios. Testing failure scenarios proactively is the only way to ensure your systems are actually resilient.

Observability is an ongoing investment. You can’t build it once and forget it. New services need instrumentation. New failure modes need monitoring. Alerts need tuning to reduce false positives. I spend 2-3 hours per week maintaining and improving our observability systems, and it’s time well spent.

Where to Start

If you’re running production systems without proper observability, here’s the path I’d recommend:

  1. Start with Prometheus for metrics (detect when things go wrong)
  2. Add centralized logging (debug why things went wrong)
  3. Implement synthetic monitoring (catch issues before users do)
  4. Graduate to chaos engineering (validate resilience and monitoring)

Don’t try to build everything at once. I spent two years building this observability stack. Start with the area that would give you the most value. If you’re constantly surprised by incidents, start with metrics and alerting. If you can detect incidents but can’t debug them, start with centralized logging.

Read through the linked guides based on your priorities. Each guide includes real configurations, real examples, and real war stories from production.

The Observability Maturity Journey

Here’s what observability maturity looks like in practice:

Level 1 - Reactive (where we started):

Level 2 - Basic Monitoring (first improvement):

Level 3 - Application Observability (major improvement):

Level 4 - Proactive (where we are now):

The journey from Level 1 to Level 4 took four years. Each level built on the previous one. You can’t skip levels. You need the foundation of basic monitoring before you can do application observability. You need good application observability before proactive testing makes sense.

For deeper dives into specific aspects of observability:

This is how you build observability for production systems. Start simple, iterate based on pain points, and gradually build comprehensive visibility into everything that matters. The investment pays off every time you detect and fix an issue before users notice.

✦ El Muhammad's Portfolio

© 2025 Aria

Instagram YouTube TikTok 𝕏 GitHub