ML Training Pipeline Design and Retraining Gates
Design ML training pipelines with leak-free features, reproducible runs, idempotent DAGs, and champion-vs-challenger promotion gates.
Before & after
Same prompt, same agent — the only difference is the skill.
Moving from manual retraining to a production ML pipeline is a big step forward. Here's what a production-grade pipeline typically includes: - **Experiment tracking** (MLflow, Weights & Biases) to log runs - **Orchestration** (Airflow, Kubeflow) to schedule and manage workflows - **Feature store** (Feast) for consistent features across training and serving - **Model registry** to version and stage models - **Automated retraining and validation** with quality gates Start by adding experiment tracking so you have visibility, then move to orchestration for automation. What does your current setup look like?
"Breaks constantly" usually means one of three specific failures, and
the fix order matters — automate on a broken foundation and you've
automated the breakage.
FIRST, WHICH BREAK IS IT?
Train/serve skew (features differ between the two) → feature store
Silent quality regression (bad model ships) → validation gate
Manual step forgotten → orchestration
Most "constantly breaking" pipelines are the first. Diagnose before
building.
BUILD ORDER — deliberately not alphabetical
1. EXPERIMENT TRACKING (MLflow / W&B) — do this first, always
You can't fix what you can't see. Log every run's params, metrics,
data version. This is one afternoon and it makes every later step
debuggable.
2. FEATURE STORE (Feast) — if it's train/serve skew
One definition of each feature, used by both training and serving.
This is what kills "works in training, wrong in prod." Schema the
features here, not in two places.
3. MODEL REGISTRY + VALIDATION GATE
A model is promoted staging→prod ONLY if it beats the current one
on held-out data. This gate is what stops a bad retrain from
shipping — the "breaks constantly" you described is often a bad
model auto-deploying with no gate.
4. ORCHESTRATION (Airflow / Kubeflow DAG)
NOW automate — retrain → validate → register → deploy, with the
gate wired in. Automating before step 3 just ships bad models
faster.
5. DATA VERSIONING (DVC)
So a retrain is reproducible: same data version + same code = same
model. Without it, "it worked last week" is unfalsifiable.
THE PRINCIPLE
Visibility → correctness → automation. In that order. The pipeline
that breaks constantly almost always automated before it had the
validation gate.
What's the actual failure — wrong predictions in prod, or a step that
falls over? That picks whether we start with Feast or the gate.About this skill
Your model scored great offline and fell over in production. Or nobody can say which data trained the model that is serving right now. Or the weekly retrain shipped a worse model because nothing compared it to the old one. This skill designs and reviews ML training pipelines so those failures get caught before they ship.
What it does
- Finds label leakage and point-in-time join bugs in feature pipelines, including Feast historical retrieval and hand-rolled SQL
- Picks a retraining trigger (schedule, drift, or performance) that fits how long your labels take to arrive
- Writes idempotent Airflow, Prefect, or Kubeflow DAGs with partitioned outputs, retries for transient errors only, and no hidden
now()calls - Makes every run reproducible by logging the code SHA, data snapshot hash, config, environment lock, and seed to MLflow or Weights & Biases
- Builds a champion-vs-challenger promotion gate with a frozen eval set, bootstrap confidence intervals, slice regression limits, and alias-based rollback
- Right-sizes the stack, and tells you when you do not need a feature store or Kubeflow yet
What you get
A pipeline design with a stage table, DAG code, the exact promotion gate logic, and the lineage fields to log. If you paste existing code, you get a review that ranks the leakage and reproducibility risks with a fix for each.
Best for
ML engineers and data scientists moving a model from notebooks to scheduled production training, and platform engineers reviewing a team's MLOps setup.
SKILL.md preview
---
name: ml-training-pipeline-retraining-gates
description: Designs and reviews production ML training pipelines - point-in-time-correct features, time-aware splits, idempotent orchestration DAGs (Airflow, Prefect, Kubeflow), reproducible run lineage in MLflow or W&B, and champion-vs-challenger promotion gates with rollback. Use when the user says "set up a retraining pipeline for our model", "our model is great offline but bad in production", "build an Airflow DAG for training", "do we need a feature store", "I can't reproduce last month's model", "automated retraining shipped a worse model", "review my MLflow / Feast setup", or "my AUC is suspiciously high".
---
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Data Wrangling Without Silent Errors
Correct, fast pandas code for merging, cleaning, groupby, reshaping, and time series, with checks that catch silent row loss.
Apache Spark Job Tuning and Debugging
Finds why a Spark job is slow or failing from the plan and UI metrics, then gives the code and config fix plus how to verify it.
Reliable Data Pipelines and Analytics Models
Build and repair ETL/ELT pipelines with replay-safe writes, explicit data contracts, and actionable quality checks.
Statistical Tests, Power & A/B Readouts
Get hypothesis tests, effect sizes, confidence intervals, and sample-size calculations with experiment validity checks.