RAG Pipeline Design and Retrieval Debugging
Design or fix a RAG pipeline: chunking, hybrid search, reranking, and a retrieval eval that pinpoints which stage is failing.
Before & after
Same prompt, same agent — the only difference is the skill.
I can help you build a production RAG system from scratch. First, you'll need to chunk your documents into smaller pieces, generate embeddings for those chunks, and set up a vector database to store them. Then you'd build a search pipeline that retrieves relevant document chunks when a user asks a question. There are several different vector database options available to choose from. What technology stack are you currently using in your infrastructure? Do you want to include hybrid search combining keyword and semantic matching?
PRODUCTION RAG ARCHITECTURE SPECIFICATION: CHUNKING STRATEGY: Recursive text splitter with 512 token chunks as default size, 100 token overlap between chunks. Preservation rule: never split within code examples, tables, or images. Include metadata tags: page_title, section_name, update_date, doc_version, author. Chunk quality validation. EMBEDDINGS MODEL: OpenAI text-embedding-3-small (1536 vector dimensions). Cost approximately 0.02 USD per 1 million tokens. Regenerate full embedding index monthly to capture new docs and updates. Batch processing runs every Sunday 2 AM UTC automatically. VECTOR STORE INFRASTRUCTURE: Postgres database with pgvector extension installed. Self-hosted deployment, no third-party vendor dependency. Index type: HNSW (Hierarchical Navigable Small World) for 99th-percentile recall accuracy. Automatic reindex trigger after 10,000 new embeddings accumulated. HYBRID SEARCH LOGIC: BM25 keyword matching combined with vector semantic similarity. Fusion weighting: 60% semantic similarity score, 40% keyword matching score. Reranker: Cohere rerank-3-english model, re-scores top 20 results before returning top 5 to end user. RETRIEVAL PIPELINE FLOW: User query input → Query rewriting for clarity (LLM, under 1 second latency) → Hybrid search execution (retrieve 20 candidates) → Rerank candidates → LLM answer generator → Response delivery to user. EVALUATION & MONITORING: Quarterly RAGAS benchmark evaluation on 100 held-out queries. Track precision (target 0.85), recall (target 0.80), F1 score (target 0.75). Adjust chunk size and overlap parameters if metrics drift below targets.
About this skill
Your RAG demo worked on ten documents. Now it returns plausible but wrong chunks, misses exact part numbers, and answers from last year's policy, and nobody can say whether chunking, embeddings, or the prompt is to blame. This skill finds the broken stage first, then fixes that stage with concrete settings and a small labeled eval that proves the fix worked.
What it does
- Sorts failures into four buckets (not ingested, not retrieved, retrieved but ranked too low, retrieved but misused by the model) so you stop tuning the wrong knob
- Specifies structure-aware chunking with breadcrumb headers, whole tables and code blocks, and parent-child retrieval where it helps
- Designs hybrid BM25 plus dense retrieval fused with Reciprocal Rank Fusion, so IDs, error codes, and names stop slipping through
- Sizes the reranking step (candidate count, final k, latency cost) and applies tenant and permission filters inside the query, not after it
- Builds a 50-200 query retrieval eval with recall@k and MRR, scored separately from answer faithfulness
- Flags when RAG is the wrong tool: counting or aggregate questions, structured data, or corpora small enough to fit in context
What you get
For a new system, a pipeline spec with a stated default for every stage plus an eval plan. For a failing system, a stage-by-stage diagnosis, ranked fixes, and the test that confirms each one.
Best for
Engineers building or maintaining knowledge-base chatbots, support assistants, internal search, or document Q&A on any vector store or framework.
SKILL.md preview
---
name: rag-pipeline-design-and-retrieval-debugging
description: Designs new RAG pipelines and diagnoses failing ones by isolating the broken stage (parsing, chunking, embedding, hybrid retrieval, reranking, context assembly, generation), fixing that stage, and proving the fix with a labeled retrieval eval. Use when the user says "design a RAG system for our docs", "what chunk size should I use", "my RAG keeps returning irrelevant chunks", "the bot can't find exact part numbers or error codes", "should I add a reranker", "how do I evaluate retrieval", "my chatbot answers from the outdated version of the doc", "which vector database should we use".
---
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Data Wrangling Without Silent Errors
Correct, fast pandas code for merging, cleaning, groupby, reshaping, and time series, with checks that catch silent row loss.
Apache Spark Job Tuning and Debugging
Finds why a Spark job is slow or failing from the plan and UI metrics, then gives the code and config fix plus how to verify it.
Reliable Data Pipelines and Analytics Models
Build and repair ETL/ELT pipelines with replay-safe writes, explicit data contracts, and actionable quality checks.
Statistical Tests, Power & A/B Readouts
Get hypothesis tests, effect sizes, confidence intervals, and sample-size calculations with experiment validity checks.