Senior Data Engineer
Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.
Before & after
Same prompt, same agent — the only difference is the skill.
Real-time analytics pipelines are definitely doable, though there are quite a few architectural decisions to make. You'd want to think about whether you're using something like Kafka for streaming, what your data warehouse looks like, whether to use Spark or Flink for transformations, and how you're handling schema evolution. Also depends on your scale and latency requirements. Are we talking thousands of events per second or millions? How fresh does the data need to be? Do you have existing infrastructure in AWS or GCP? Once I understand your constraints better, I can recommend a specific tech stack and walk you through the design.
REAL-TIME ANALYTICS PIPELINE ARCHITECTURE INGESTION LAYER • Event producers: Web SDK, mobile SDK, backend services • Kafka cluster (3 brokers) for event buffering → Topic: events_raw with 12 partitions → Retention: 7 days (500GB uncompressed daily volume) • Schema Registry enforcing Avro schemas per event type STREAM PROCESSING (Apache Spark Streaming) • 30-second micro batches for user_session aggregations • Transformations: deduplication, timestamp validation, IP geolocation • Sliding window (5-minute) for active-user counts and session metrics • Output to intermediate Kafka topic (events_enriched) DATA WAREHOUSE (Snowflake) • events_raw table (1.2B rows/day, partitioned by date) • user_sessions table (15M rows/day, includes user_id, session_id, duration, events_count) • Real-time dashboard syncs from _enriched topic via Fivetran MONITORING • Alert if processing lag exceeds 2 minutes • Track schema validation failures per event type • Monitor Kafka consumer group lag (target: <1 minute) COST: ~$8k/month (compute, storage, orchestration combined)
About this skill
name: senior-data-engineer description: Use when Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.
Senior Data Engineer
Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.
What you get
- Public GitHub repo (alirezarezvani/claude-skills)
- the senior-data-engineer skill folder with SKILL.md. Part of a 337-skill / 30-agent / 70-command install.
Customize your output
- Fork the repo and adapt the skill's instructions and references to your workflow.
Example output
Activates automatically when your request matches Senior Data Engineer; chains with the other skills, agents, and commands in the collection.
Best for
Creators, builders, and teams using Claude Code.
SKILL.md preview
---
name: senior-data-engineer
description: Use this skill when designing data architectures, building pipelines, optimizing data workflows, or troubleshooting data quality and infrastructure issues.
version: 1.0.0
category: Development / Engineering
author: AgentVolt
license: proprietary
tags:
- development
- engineering
---
# Senior Data Engineer
Applies production data-engineering practice to pipeline design, orchestration, and data quality across the modern Python/SQL/Spark stack.
## When to use
… (sign up to view the full skill)More development skills
View all Development skills →Generate
Scaffold code, tests, and boilerplate from a short spec.
Performance Profiler
Systematic performance profiling for Node.js, Python, and Go applications.
Threat Detection
Use when hunting for threats in an environment, analyzing IOCs, or detecting behavioral anomalies in telemetry.
Agent Harness
Turns a domain folder of skills into a bounded agentic loop: compile a goal into a task plan, execute with the domain's tools, verify every task, retry within a budget.