Service Observability and SLO Design
Get a service observability plan with precise SLIs, burn-rate alerts, correlated telemetry, and a staged rollout.
Before & after
Same prompt, same agent — the only difference is the skill.
I can help you design an observability strategy for your new microservice. There are three pillars you'll want to consider: metrics, logs, and traces. Metrics give you the aggregate view, logs help with debugging, and traces show you the flow of requests across services. You might want to start by identifying your key business metrics and technical metrics. I could walk you through the golden signals framework or help you think about SLO design. What's your current observability stack—Prometheus, Datadog, something else? Once I understand your setup, I can help you design a strategy that fits your needs.
OBSERVABILITY STRATEGY FOR MICROSERVICE SERVICE DETAILS • Name: OrderProcessor (payment flow) • Criticality: Tier 1 (revenue-impacting) • Traffic: 50 req/sec (peak 120 req/sec) GOLDEN SIGNALS LATENCY • Metric: p50, p95, p99 response time (ms) • Target: p95 < 200ms • Alert: p95 > 250ms for 2 min ERROR RATE • Metric: 5xx errors / total requests • Target: < 0.1% (0.001 error rate) • Alert: > 0.5% for 1 min TRAFFIC • Metric: requests per second • Target: normal range 40-80 req/sec • Alert: < 10 req/sec (possible outage) or > 150 req/sec (spike) SATURATION • Metric: CPU (% used), Memory (%), Database connections (active/pool size) • Target: CPU < 60%, Memory < 70% • Alert: CPU > 80% or Memory > 85% SLI/SLO DEFINITION Service Level Indicator: Successful requests (HTTP 2xx) / Total requests • Good: response < 200ms AND status 2xx • Bad: timeout OR 5xx error • Window: 30-day rolling Service Level Objective: 99.5% (99.5% of requests succeed) → Budget: 3.6 minutes of downtime per 30 days → Used this month: 1.2 minutes (67% of budget remaining) LOG LEVELS & RETENTION • ERROR: Failed payment attempt, database timeout, circuit breaker open → 30 days • WARN: Retry attempt, slow query (>500ms) → 7 days • INFO: Request received, order processed, payment confirmed → 3 days • DEBUG: SQL queries, stack traces → 1 day (dev only) TRACES • Instrumentation: OpenTelemetry SDK • Sample rate: 10% (high-volume), 100% for errors • Propagate trace IDs across services (HTTP headers) • Key spans: database call, payment gateway call, response ALERTS (Escalation: Slack → PagerDuty if p1) Severity: P1 (immediate) • Error rate > 1% for 1 minute • Latency p95 > 500ms for 3 minutes → Action: Page on-call engineer Severity: P2 (investigate within 1 hour) • Error rate > 0.5% for 5 minutes • Memory usage > 85% → Action: Slack alert to team Severity: P3 (track for next sprint) • Database connection pool nearing capacity • Warn logs > 10 per second → Action: Logged for retro review
About this skill
“Our dashboards look healthy, but customers are still failing.” “Everything pages, so nobody trusts the alerts.” This skill turns those problems into a concrete observability design: measurable user outcomes, correlated metrics/logs/traces, and alerts tied to a response someone can take.
What it does
- Defines SLI numerators, denominators, eligibility rules, measurement points, and missing-data behavior.
- Calculates error budgets and multiwindow burn-rate alerts, including explicit handling of low traffic.
- Maps critical request paths and asynchronous jobs to metrics, structured logs, and boundary spans.
- Reviews existing alerts and assigns a keep, rewrite, ticket, dashboard, or retire decision with a reason.
- Controls telemetry cost through label budgets, sampling choices, retention priorities, and sensitive-data exclusions.
- Specifies failure-injection checks for alert timing, missing telemetry, duplicate pages, and request correlation.
What you get
A scoped design or alert review with precise signal definitions, routing decisions, implementation priorities, and acceptance checks. When your stack is known, it can also produce configuration or instrumentation code with assumptions and unverified details clearly marked.
Best for
Backend engineers, platform teams, and on-call owners adding observability to a service, repairing noisy paging, or introducing an SLO program.
SKILL.md preview
---
name: service-observability-and-slo-design
description: "Design service observability using user-journey SLIs, explicit error-budget math, correlated metrics/logs/traces, and actionable alert routing. Use when the user says 'add observability to this service', 'our alerts are too noisy', 'my dashboards are green but customers are failing', 'design SLIs and SLOs', 'review these monitoring rules', 'what should we instrument first', or 'our tracing bill keeps growing'."
---
# Service Observability and SLO Design
… (sign up to view the full skill)More development skills
View all Development skills →Chaos Experiment and Game Day Planner
Plans chaos experiments and game days: hypothesis, blast radius, abort conditions, tool manifests, rollback, and post-run review.
Claude Code Cost and Latency Routing
Configure Claude Code Router with model tiers, explicit fallbacks, compatibility checks, and a cost-aware validation plan.
CI/CD Pipeline Builder with Gated Deploys
Generates CI/CD workflows from your repo's real stack: cached fast-fail checks, pinned actions, and staging-to-prod deploys with gates.
Terraform Modules and Safe Deployments
Get reusable Terraform modules, safe state migrations, explicit provider wiring, and CI gates tied to reviewed plans.