Production ML Deployment and Reliability
Ship models, harden RAG and LLM APIs, and diagnose production drift with release gates, rollback plans, and cost controls.
Before & after
Same prompt, same agent — the only difference is the skill.
Model drift monitoring is important for maintaining model performance over time. You'll want to set up some kind of tracking for how your model's predictions compare to actual outcomes, and alert if there's significant degradation. You might use tools like MLflow or Evidently AI to help with this. You should probably define what drift means for your use case—whether it's prediction shift, feature shift, or performance degradation. You'd want to collect baseline metrics from when the model was trained and compare current performance to those baselines. It might help to set up automated alerts when performance drops below a threshold. You could also periodically retrain your model on fresh data. Would you like guidance on which monitoring tool to use, or help defining drift thresholds for your specific problem?
MODEL DRIFT MONITORING SETUP DEFINE DRIFT TYPES → Data drift: Feature distributions changed (e.g., user demographics shift) → Prediction drift: Model output distribution drifts (e.g., predictions skewed to one class) → Performance drift: Accuracy drops vs. production baseline (e.g., precision drops below 85%) BASELINE METRICS (from training) → Accuracy: 0.89 (Recorded: 2026-06-01) → Precision (class A): 0.91 → Recall (class A): 0.87 → Feature means + std (all 47 features logged) MONITORING PIPELINE → Tool: Evidently AI (open source) or Arize (managed) → Cadence: Daily checks at 2am UTC → Sample: Latest 1,000 predictions + actuals → Metrics computed: KS test for distribution shift, current accuracy vs. baseline ALERT THRESHOLDS → CRITICAL (page on-call): Accuracy drops >5% (below 0.84) → WARNING: Any feature drifts >3 std from baseline → INFO: Prediction distribution shift >10% (logged, not alerted) RESPONSE PLAYBOOK CRITICAL ALERT → Investigate root cause (feature drift? label corruption? concept drift?) → If fixable (bad feature): Hot-patch feature pipeline → If model issue: Trigger retraining on last 30 days data → If data issue: Quarantine predictions, rollback to previous model version IMPLEMENTATION → Dashboard: Shows current metrics vs. baseline, drift score over time → Logging: All predictions logged to data warehouse for audit → Cost: ~200/month for monitoring service + retraining compute
About this skill
Your model passed offline evaluation, but production predictions changed, latency climbed, or nobody knows whether rollback is safe. This skill turns those problems into concrete deployment changes, diagnostic checks, and release gates, including the retrieval and API failures that make RAG and LLM products unreliable.
What it does
- Packages model artifacts, preprocessing, feature contracts, and serving configuration into a reproducible release with a compatible rollback path.
- Chooses batch or online inference from freshness, latency, traffic, and cost requirements without defaulting to Kubernetes or a feature store.
- Investigates performance drops by separating broken telemetry, feature skew, population shifts, and measured quality loss.
- Evaluates RAG retrieval separately from answer generation, with document permissions, citations, and index migrations accounted for.
- Bounds LLM retries, concurrency, token use, and fallback behavior against request deadlines and spending limits.
- Defines release checks, monitoring signals, and operational responses using the evidence available.
What you get
A scoped implementation, incident diagnosis, or production plan with explicit assumptions, measurable acceptance criteria, and a rollback procedure. When you provide code or configuration, the response targets the actual changes needed and distinguishes proposed checks from executed results.
Best for
ML engineers and product teams moving trained models, RAG systems, or LLM features into production, or repairing reliability problems after launch.
SKILL.md preview
---
name: production-ml-deployment-and-reliability
description: >-
Productionize trained models, diagnose serving and drift failures, and harden RAG and LLM integrations using versioned release contracts, isolated evaluations, bounded inference, and compatible rollback paths. Use when the user says "deploy this model", "set up our MLOps pipeline", "our model accuracy keeps dropping", "training and serving predictions disagree", "our RAG answers keep missing the right documents", "LLM retries are driving up our bill", or "can we safely roll back this model?".
---
# Production ML Deployment and Reliability
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Data Wrangling Without Silent Errors
Correct, fast pandas code for merging, cleaning, groupby, reshaping, and time series, with checks that catch silent row loss.
Apache Spark Job Tuning and Debugging
Finds why a Spark job is slow or failing from the plan and UI metrics, then gives the code and config fix plus how to verify it.
Reliable Data Pipelines and Analytics Models
Build and repair ETL/ELT pipelines with replay-safe writes, explicit data contracts, and actionable quality checks.
Statistical Tests, Power & A/B Readouts
Get hypothesis tests, effect sizes, confidence intervals, and sample-size calculations with experiment validity checks.