Production Observability: Alerts, Dashboards, Traces

MONITORING STACK SETUP

ARCHITECTURE OVERVIEW
→ Prometheus: Scrapes metrics every 15 seconds
→ Grafana: Visualizes metrics, triggers alerts
→ Application: Instrumented with Prometheus client library
→ Structured logs: JSON logs → centralized storage (Loki optional)

STEP 1: PROMETHEUS CONFIG
File: prometheus.yml
→ Scrape interval: 15 seconds
→ Global timeout: 10 seconds
→ Targets: localhost:8080 (your app), localhost:9090 (node exporter)

Example scrape config:
COPY TO YOUR SETUP
Scrape job 1: application metrics (port 8080)
Scrape job 2: system metrics via node-exporter (port 9100)
Retention: 30 days (disk: ~50GB estimate)

STEP 2: APPLICATION INSTRUMENTATION
Library: Add to code: prometheus client for your language

Metrics to track (high value):
→ http_requests_total (counter: request count by endpoint, method, status)
→ http_request_duration_seconds (histogram: latency by endpoint)
→ database_query_duration_seconds (histogram: query slowness)
→ cache_hits_total (counter: cache efficiency metric)
→ errors_total (counter: by error type)
→ active_connections (gauge: current open connections)

Code example (Python):
from prometheus_client import Counter, Histogram, Gauge
http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])

@app.route('/api/users')
def get_users():
  with http_duration.labels(endpoint='get_users').time():
    result = fetch_users()
    http_requests.labels(method='GET', endpoint='get_users', status='200').inc()
    return result

STEP 3: GRAFANA DASHBOARDS

DASHBOARD 1: System Health (for on-call engineers)
→ Panels:
  • CPU usage % (gauge, alert if > 80%)
  • Memory usage % (gauge, alert if > 85%)
  • Disk usage % (gauge, alert if > 90%)
  • Network IO (bytes in/out)
→ Layout: 4 panels top row
→ Refresh: 30 seconds

DASHBOARD 2: Application Performance
→ Panels:
  • Request rate (graph: requests/sec, colored by status code)
  • Response time P50/P95/P99 (graph: milliseconds)
  • Error rate % (graph: errors/total, red alert line at 5%)
  • Database query slowness (graph: slow queries in real-time)
  • Cache hit ratio % (gauge)
→ Drill-down: Click endpoint → see detailed latency breakdown

DASHBOARD 3: Business Metrics (for PMs)
→ Panels:
  • Active users (connected sessions gauge)
  • Signup events (counter, graph over time)
  • API calls by customer segment
  • Revenue impact (if tracking via metric tags)

STEP 4: ALERTING RULES

Alert Rule 1: High error rate
→ Condition: error_rate > 5% for 2 minutes
→ Action: Send to Slack #alerts channel
→ Notification: "Production: 7% errors detected. Check dashboard link."

Alert Rule 2: Slow database queries
→ Condition: p95 query duration > 500ms for 5 minutes
→ Action: Page on-call engineer
→ Notification: "Database performance degraded. Typical query: 650ms."

Alert Rule 3: High memory consumption
→ Condition: memory usage > 85% for 3 minutes
→ Action: Send to Slack
→ Escalate to: Page engineer if sustained 10 minutes

Alert Rule 4: Service unreachable
→ Condition: failed scrapes for 2 consecutive attempts
→ Action: Page on-call immediately
→ Notification: "CRITICAL: API service not responding."

STEP 5: STRUCTURED LOGGING

Log everything as JSON:
{
  "timestamp": "2024-07-31T14:22:15Z",
  "level": "ERROR",
  "service": "api",
  "message": "Database connection timeout",
  "trace_id": "abc-123-def",
  "user_id": 5047,
  "endpoint": "/api/users",
  "duration_ms": 2150
}

Benefit: Grep-able, queryable, connects to distributed traces

STEP 6: DEPLOYMENT
→ Docker compose file included (prometheus + grafana)
→ Or: Kubernetes manifests (if using K8s)
→ Startup: docker-compose up -d
→ Access Grafana: localhost:3000 (default admin/admin)

VERIFICATION CHECKLIST
☐ Prometheus scraping metrics (check /targets endpoint)
☐ At least 1 custom metric from your app (visible in Grafana)
☐ 1 dashboard created with 3+ panels
☐ 2 alert rules configured
☐ Alerts tested (trigger manually, confirm Slack delivery)
☐ On-call schedule linked to alerts

NEXT STEPS (not included in this setup)
→ Distributed tracing (Jaeger, optional)
→ Log aggregation (Loki, ELK stack)
→ Load testing (k6, Artillery) to establish baseline
→ Runbook linking (document what to do per alert)

ESTIMATED SETUP TIME
Prometheus config: 15 minutes
Instrument app: 1-2 hours (depends on codebase size)
Grafana dashboards: 1-2 hours (template provided)
Alerts: 30 minutes
Testing: 30 minutes
TOTAL: 4-6 hours for working monitoring system