Apache Spark Job Tuning and Debugging
Finds why a Spark job is slow or failing from the plan and UI metrics, then gives the code and config fix plus how to verify it.
Before & after
Same prompt, same agent — the only difference is the skill.
A 1TB query can be slow for many reasons. I'd need to understand more about your current performance—what's the actual runtime, and is it the read or compute step that's bottlenecking? What transformations are in your query? Are you partitioned, and if so by what column? Do you have skew on any key? I can help you think through parallelization and caching strategies, but it depends on your data shape. Can you share the query and current execution plan?
SPARK OPTIMIZATION PLAN: 1TB QUERY DIAGNOSIS (From explain plan) Stage 1 → Full table scan (no partition pruning) - 45% runtime Stage 2 → Wide shuffle on user_id (data skew detected) - 40% runtime Stage 3 → Sort (unnecessary, can optimize order) - 15% runtime OPTIMIZATIONS PARTITIONING • Repartition table by date column (yearly buckets) • Add partition predicate → WHERE date >= '2024-01-01' • Result → 70% of scans eliminated in future runs SHUFFLE REDUCTION • Broadcast the smaller dimension table (events) • Use broadcast join instead of sort-merge • Handle data skew → Split top 100 user_ids into separate join paths • Result → Shuffle size cut from 500GB to 200GB CACHING • Cache intermediate aggregation → .cache() before wide transform • Save results to Parquet (columnar, snappy compression) • Result → Re-runs 80% faster EXECUTION CONFIG Executor memory → 8GB per executor (vs. default 4GB) Shuffle partitions → 400 (vs. default 200) Broadcast threshold → 500MB (vs. default 10MB) EXPECTED GAINS Current runtime: 47 minutes Optimized runtime: 12-14 minutes (70% improvement) Cost savings: 3-4x executor hours reduced
About this skill
Your Spark job has 1,999 tasks that finish in a minute and one that runs for 45. Or executors keep dying with "container killed for exceeding memory limits." Or a job writes 80,000 tiny Parquet files. Doubling the cluster didn't help. This skill diagnoses Spark problems the way a performance engineer does. It reads the physical plan and stage metrics, names the mechanism, and changes the fewest things that fix it.
What it does
- Reads Spark UI stage metrics to separate the common causes: key skew, undersized partitions, memory pressure, small files, and driver-side bottlenecks. It compares median and max task times, shuffle read, spill, and GC time.
- Reads
explain("formatted")output to find shuffles, missed broadcasts, empty pushed filters, Python UDFs, and accidental cartesian joins. - Fixes skewed joins in a deliberate order: AQE skew handling first, then null and sentinel key isolation, then targeted salting, with working PySpark or Scala code.
- Sizes shuffle partitions from data volume rather than the 200 default, and accounts for what AQE already does on Spark 3.x.
- Maps each memory error to its cause: YARN/K8s container kills, executor heap OOM, driver OOM, and FetchFailed. It raises the right setting instead of always raising heap.
- Reviews PySpark and Spark SQL code for costly anti-patterns:
withColumnloops, row-at-a-time UDFs,coalesce(1), unpartitioned windows, and caching data that is used only once.
What you get
You get a diagnosis with the evidence behind it and a concrete code and config fix. You also get the metric that should move and where to check it, plus the next hypothesis if the first one doesn't pan out.
Best for
Data engineers and analytics engineers running PySpark, Scala Spark, or Spark SQL on Databricks, EMR, Dataproc, Kubernetes, or YARN who need a slow or crashing job fixed, not a list of config options.
SKILL.md preview
---
name: apache-spark-job-tuning-and-debugging
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Data Wrangling Without Silent Errors
Correct, fast pandas code for merging, cleaning, groupby, reshaping, and time series, with checks that catch silent row loss.
Reliable Data Pipelines and Analytics Models
Build and repair ETL/ELT pipelines with replay-safe writes, explicit data contracts, and actionable quality checks.
Statistical Tests, Power & A/B Readouts
Get hypothesis tests, effect sizes, confidence intervals, and sample-size calculations with experiment validity checks.
Answer Business Questions with Validated SQL and Charts
Turn a business question into a pinned metric definition, checked SQL, the right chart, and a plain finding with confidence and caveats.