Building Production MLOps Infrastructure: From Drowning Detection to Distributed Training at Scale

Jan 20, 2025

I’ve spent the last five years building production machine learning systems, and it’s been nothing like the tutorials make it seem. The tutorials show you how to train a model in a Jupyter notebook. They don’t show you how to deploy that model to edge devices with sub-500ms latency requirements. They don’t show you what happens when your training pipeline takes 18 days and your data scientists are bottlenecked. They definitely don’t show you what it takes to build a computer vision system that needs to detect drowning in real-time while running on limited hardware at the edge of a pool.

Let me walk you through three major ML systems I’ve built, each teaching me different lessons about what it takes to run machine learning in production.

System 1: AI-Powered Drowning Detection (Edge ML at Scale)

The Problem: I built a drowning detection system that had to work in the real world, not in a lab. Real pools with reflections, splashing, kids playing, adults swimming laps. The system needed to tell the difference between normal pool activity and actual drowning. And it needed to do this in real-time, processing video feeds with under 500ms latency, because every second matters when someone is drowning.

The Wake-Up Call: During early testing, I deployed a prototype to a test pool and watched it for a week. The false positive rate was brutal. Kids doing handstands underwater triggered alerts. Adults diving to retrieve pool toys triggered alerts. Even heavy splashing from normal play sometimes triggered alerts. I had built a technically impressive model with 94% accuracy in the lab, but in production it was causing alert fatigue that would make the system useless.

The Solution: I built a hybrid edge-cloud architecture specifically designed for real-time computer vision at the edge. The critical inference (is this drowning?) runs on edge devices at the pool using optimized TensorFlow Lite models. These edge devices process video feeds locally with sub-500ms latency and can function even if internet connectivity is lost.

The cloud side handles model training, evaluation, and distribution. I used Google Kubernetes Engine to run the training pipeline. New models are trained on thousands of hours of pool video (collected with proper consent and privacy controls), validated extensively, then pushed to edge devices through an automated deployment pipeline.

The data pipeline was crucial. Every time the edge system makes a prediction, it logs the video frames and prediction confidence. If an alert fires (real or false positive), that gets flagged for review. This creates a continuous feedback loop where the model gets better over time from real-world data.

I also built an alert escalation system with multiple confidence thresholds. A low-confidence detection triggers a silent alert to lifeguards’ devices. A medium-confidence detection triggers an audible alert. A high-confidence detection triggers all alarms and logs. This reduced false positive fatigue while ensuring real incidents got immediate attention.

What Made This Hard:

The edge deployment constraints were brutal. I couldn’t use a massive GPU-hungry model. I was running on ARM processors with limited memory. I spent weeks optimizing the model, using techniques like quantization, pruning, and knowledge distillation to get the model small enough to run on edge hardware while maintaining accuracy.

Privacy and security were non-negotiable. This system processes video of children at pools. I implemented end-to-end encryption for all video transmission. Video is processed locally on edge devices and only prediction metadata (not raw video) goes to the cloud unless explicitly flagged for review. I worked with privacy lawyers to ensure GDPR and CCPA compliance.

The failure modes were scary. What if the edge device crashes? What if it loses power? What if internet connectivity drops? I built redundancy at every level. Multiple edge devices per pool running the same model, with automatic failover. Local battery backup for edge devices. Degraded mode operation when cloud connectivity is lost.

The Results:

The system has been running in production at multiple pool facilities for over two years. The current model achieves 96.8% accuracy in real-world conditions (measured against expert lifeguard review of flagged incidents). False positive rate is down to 0.3 per pool per day, which is low enough that lifeguards don’t develop alert fatigue.

Most importantly, the system has successfully detected 7 real drowning incidents in production, all of which were responded to within seconds. In each case, the victim was rescued before serious harm occurred. This is what makes all the technical challenges worth it.

Read the full guide: AI-Powered Drowning Detection System

What you’ll learn:

System 2: Distributed Training Platform with Ray (Cutting Training Time by 10x)

The Problem: When I joined the team, data scientists were struggling with a massive bottleneck. They were trying to train deep learning models on single GPUs, waiting hours or days for results. GPU availability was a constant fight. Someone would submit a training job at 9 AM and might not get GPU access until midnight. The cloud costs were approaching $40,000 per month, and people were still bottlenecked.

The Frustration: I sat in on a weekly data science meeting where the team was reviewing the previous week’s experiments. Out of 12 experiments they wanted to run, they’d only completed 4 because of GPU availability. The other 8 were still queued or had failed midway through when someone else needed the GPU. The lead data scientist said something that stuck with me: “We’re spending more time managing infrastructure than building models.”

The Solution: I built a self-service distributed training platform on Kubernetes using Ray. Ray handles the complexity of distributed training, letting data scientists write Python code that scales across multiple GPUs without thinking about the infrastructure.

The architecture uses GKE with separate node pools for different workloads. On-demand GPU nodes for critical production training that can’t fail. Preemptible GPU nodes (70% cheaper) for experimentation and development. CPU node pools that autoscale independently for data preprocessing. This separation meant we stopped wasting expensive GPU time on tasks that didn’t need GPUs.

The autoscaling was the trickiest part to get right. My first implementation was too aggressive, spinning up 10 GPU nodes the moment a job was queued, even if it only needed 2 GPUs. I woke up to a Slack message from finance asking why we’d burned through $3,000 overnight. I rebuilt it with a smarter controller that looked at actual job resource requirements and scaled appropriately.

I also built automatic fault tolerance through checkpointing. When a preemptible node gets terminated mid-training (which Google does with zero warning), Ray automatically catches the failure and resumes from the last checkpoint on a new node. The first time I tested this by manually killing a node during a 6-hour training run, my heart was pounding. Five minutes later, the job resumed like nothing happened.

What Made This Hard:

Developer experience was critical for adoption. If the platform was hard to use, data scientists would just go back to fighting over single GPUs. I spent a week building a CLI tool that reduced job submission to a single command. I integrated everything with Jupyter notebooks because that’s where data scientists actually work. The goal was to make distributed training feel no different from running code locally.

Cost visibility and optimization required instrumentation I didn’t initially build. I added Grafana dashboards showing real-time cost per experiment. When people could see exactly how much each training run cost, behavior changed. Suddenly people were asking “how can we optimize this?” instead of just running expensive jobs blindly.

The Results:

Three months after launch, the numbers exceeded my expectations. Training time for models that previously ran on a single GPU improved by 10x. What took 2 days now finished in 4 hours. Average GPU utilization jumped from 40% to 85% (we were wasting 60% of GPU capacity before). Monthly costs dropped from $40,000 to $14,000, a 65% reduction, primarily from preemptible nodes and smarter autoscaling.

The platform became fully self-service. Data scientists could spin up distributed training jobs themselves in minutes without tickets to the infrastructure team. The team started running 3x more experiments per week, shipping ML features faster than ever. We hit 99.2% successful job completion rate even with heavy use of cheap preemptible nodes.

Read the full guide: Distributed Training Platform with Ray

What you’ll learn:

System 3: Production ML Pipeline for 3D Scene Reconstruction (44% Faster Training)

The Problem: At a previous company, I inherited a ML training pipeline that was crippling the team’s ability to innovate. Training a single model for their real-time 3D scene reconstruction system took 18 days. Eighteen days. Every model iteration took nearly three weeks, which meant the feedback loop for improving models was measured in months. The team was trying to build cutting-edge computer vision, but they were moving at a crawl.

The Technical Debt: When I dug into the existing system, I found layers of technical debt. Training ran on a hodgepodge of individual GPU servers, manually managed. Data preprocessing happened on the same machines as training, bottlenecking both. There was no fault tolerance, so if a training job crashed on day 16, you started over from scratch. The team had cobbled this together as they grew, and now the infrastructure was the bottleneck.

The Solution: I designed a hybrid cloud-edge architecture with clear separation of concerns. Cloud (GKE) for heavy training workloads. On-premise edge nodes for inference where sub-500ms latency was required. A GitOps-based deployment pipeline to automate model distribution from cloud to edge.

For the training infrastructure, I deployed Ray on GKE with elastic autoscaling. Separate node pools for GPU training and CPU data preprocessing. Terraform for infrastructure as code. Helm for deploying Ray and ML applications. The entire stack managed through GitOps (Flux), which was essential for the globally distributed team.

I optimized the data pipeline significantly. The preprocessing (which didn’t need GPUs) moved to dedicated CPU nodes that scaled independently. I parallelized data loading across multiple workers. I added caching for frequently accessed datasets. These optimizations meant GPUs were actually doing training, not waiting for data.

For the global team collaboration, GitOps was transformative. Engineers in different time zones could push infrastructure changes through Git. CI/CD pipelines would validate, test, and deploy automatically. No more manual coordination or SSH access fights.

What Made This Hard:

Balancing cloud training with edge inference required careful architecture. I couldn’t just throw everything in the cloud because latency requirements for real-time video analytics demanded edge processing. I designed a clear boundary: cloud for training and model registry, edge for inference and low-latency processing.

The hybrid architecture meant I needed automated model deployment from cloud to edge. I built a CI/CD pipeline that would train models in GKE, validate them against test datasets, push successful models to a registry, then deploy to edge nodes through GitOps. The entire flow from training to edge deployment became automated.

The Results:

The impact was dramatic. Training time went from 18 days to 7 days, a 44% reduction. This meant the team could iterate on models weekly instead of monthly. Inference latency on edge nodes stayed under 500ms even for complex 3D reconstruction. Model deployment cycle went from weeks (manual, error-prone) to hours (automated, reliable).

The distributed team could finally collaborate effectively. GitOps gave everyone reliable infrastructure access. Automated deployment meant no more “it works on my machine” problems. The team’s velocity increased noticeably as infrastructure stopped being the bottleneck.

Read the full guide: Distributed ML Pipeline for 3D Reconstruction

What you’ll learn:

The Complete MLOps Framework

Here’s how all these ML systems fit together into a production MLOps framework:

Infrastructure Layer:

Model Development:

Deployment Pipeline:

Data Pipeline:

Monitoring & Operations:

What This Actually Achieved

The numbers from five years of building production ML systems:

Training Performance:

Cost Optimization:

Developer Productivity:

Production ML Impact:

Lessons Learned From Production MLOps

Model accuracy in the lab means nothing. My drowning detection model had 94% accuracy in carefully curated test data. In production with reflections, varying light conditions, and normal pool activity, it performed much worse until I retrained on real-world data. Always test with production data, not clean academic datasets.

Edge deployment is fundamentally different from cloud deployment. You can’t assume unlimited compute, reliable connectivity, or easy updates. Design for offline operation, graceful degradation, and careful resource management. The constraints make everything harder but also force you to build more robust systems.

Cost optimization requires visibility. Data scientists don’t optimize what they can’t see. The cost-per-experiment dashboard transformed behavior. People started caring about preemptible nodes, efficient data loading, and shutting down idle resources when they could see the direct cost impact.

Self-service platforms drive adoption. The distributed training platform succeeded because data scientists could use it without understanding Kubernetes, Ray internals, or infrastructure complexity. They just ran a command and got distributed training. Hide complexity behind simple interfaces.

Fault tolerance is mandatory for long-running training. When training jobs take hours or days, failures are inevitable. Hardware fails. Preemptible nodes get terminated. Jobs crash. Automatic checkpointing and resume is the difference between “lost 17 days of work” and “lost 5 minutes of work.”

GitOps transforms team collaboration for ML. When infrastructure is code and deployments are automated through Git, distributed teams can work effectively. No more manual coordination. No more “it works on my machine.” Just commit, and the system deploys.

Privacy and security aren’t optional. The drowning detection system processes video of children. I worked with lawyers to ensure compliance. I built privacy controls from day one. You can’t retrofit privacy into a system that wasn’t designed for it.

Where to Start With MLOps

If you’re building production ML systems or planning to, here’s the path I’d recommend:

  1. Start with infrastructure for distributed training (eliminate the GPU bottleneck)
  2. Build automated deployment pipelines (get models to production reliably)
  3. Add monitoring and feedback loops (improve models from production data)
  4. Optimize for costs (ML training is expensive, visibility drives optimization)
  5. Consider edge deployment when latency or privacy demands it

Don’t try to build everything at once. I spent five years learning these lessons. Start with the bottleneck that’s causing the most pain right now.

Read through the linked guides based on your priorities. Each guide includes real architectures, real code, real costs, and real war stories from production ML systems.

The Reality of Production ML

Running machine learning in production is fundamentally different from training models in notebooks. You need robust infrastructure that handles failures gracefully. You need deployment pipelines that are reliable and fast. You need monitoring that tells you when models degrade. You need cost controls that prevent runaway spending. You need privacy and security baked in from the start.

The technical challenges are significant. The operational complexity is higher than most people expect. You will have incidents. Models will behave unexpectedly in production. Infrastructure will fail. I’ve experienced all of this.

But when it works, when you build ML systems that actually solve real problems for real users, it’s incredibly rewarding. I built a system that has helped prevent drowning incidents. I built platforms that let data scientists innovate 3x faster. I’ve cut training times from weeks to days and costs from $40k to $14k per month.

This is what production MLOps looks like. Not the tutorials. The real thing, with all its complexity, challenges, and rewards.

For deeper dives into supporting infrastructure for ML systems:

This is the knowledge I wish I had when I started building production ML systems. I hope it helps you avoid some of the mistakes I made and build reliable MLOps infrastructure faster.

✦ El Muhammad's Portfolio

© 2025 Aria

Instagram YouTube TikTok 𝕏 GitHub