Spark Engineer
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads.
Before & after
Same prompt, same agent — the only difference is the skill.
A 1TB query can be slow for many reasons. I'd need to understand more about your current performance—what's the actual runtime, and is it the read or compute step that's bottlenecking? What transformations are in your query? Are you partitioned, and if so by what column? Do you have skew on any key? I can help you think through parallelization and caching strategies, but it depends on your data shape. Can you share the query and current execution plan?
SPARK OPTIMIZATION PLAN: 1TB QUERY DIAGNOSIS (From explain plan) Stage 1 → Full table scan (no partition pruning) - 45% runtime Stage 2 → Wide shuffle on user_id (data skew detected) - 40% runtime Stage 3 → Sort (unnecessary, can optimize order) - 15% runtime OPTIMIZATIONS PARTITIONING • Repartition table by date column (yearly buckets) • Add partition predicate → WHERE date >= '2024-01-01' • Result → 70% of scans eliminated in future runs SHUFFLE REDUCTION • Broadcast the smaller dimension table (events) • Use broadcast join instead of sort-merge • Handle data skew → Split top 100 user_ids into separate join paths • Result → Shuffle size cut from 500GB to 200GB CACHING • Cache intermediate aggregation → .cache() before wide transform • Save results to Parquet (columnar, snappy compression) • Result → Re-runs 80% faster EXECUTION CONFIG Executor memory → 8GB per executor (vs. default 4GB) Shuffle partitions → 400 (vs. default 200) Broadcast threshold → 500MB (vs. default 10MB) EXPECTED GAINS Current runtime: 47 minutes Optimized runtime: 12-14 minutes (70% improvement) Cost savings: 3-4x executor hours reduced
About this skill
name: spark-engineer description: Use when Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads.
Spark Engineer
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
What you get
- Public GitHub repo
- the skills/spark-engineer folder with SKILL.md and references.
Customize your output
- Fork the repo and extend the skill's reference files for your own stack conventions.
Example output
Activates on a matching request (e.g. building or reviewing Spark Engineer code) and can chain with other skills in the pack.
Best for
Full-stack developers and engineering teams using Claude Code.
SKILL.md preview
---
name: spark-engineer
description: Use this skill when writing Spark jobs, debugging performance issues, or configuring cluster settings for distributed data processing pipelines.
version: 1.0.0
category: Development / Data & ML
author: AgentVolt
license: proprietary
tags:
- development
- data-ml
---
# Spark Engineer
Handles Apache Spark work end to end: DataFrame transformations, tuning shuffle-heavy jobs, and cluster configuration matched to the workload.
## When to use
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Pro
Performs pandas DataFrame operations for data analysis, manipulation, and transformation.
Fine Tuning Expert
Use when fine-tuning LLMs, training custom models, or adapting foundation models for specific tasks.
RAG Architect
Designs and implements production-grade RAG systems by chunking documents, generating embeddings, configuring vector stores, building hybrid search pipelines.
Database Optimizer
Optimizes database queries and improves performance across PostgreSQL and MySQL systems.