Data & Analytics Skill
Airflow DAG Patterns
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and…
$15
Spark Optimization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning.
A production playbook for making slow Apache Spark jobs fast and cheap. It attacks the real bottlenecks: shuffle, data skew, partition sizing, and memory pressure: with concrete PySpark patterns, broadcast and bucket join strategies, and an AQE-enabled configuration template so your pipelines scale without exploding cluster costs.
$15 one-time
Add to a kit → Prices include 20% VAT. · Forged on real agency work · one-time, no lock-in
- Type Skill
- Category Data & Analytics
- Delivery Email · instant
- License One-time
Inside the run · no black box
See the actual work before you buy it.
The diagnosis order the skill follows on a slow Spark job, most expensive cost first:
- Open the Spark UI and find the stage that dominates wall time; read the task duration histogram for skew. A max-to-average partition ratio above 2x means one hot partition is holding the whole job hostage.
- Hunt shuffles first, because shuffle is the most expensive operation in Spark: swap repartition for coalesce where partition count only shrinks, pre-aggregate locally before groupBy, and replace exact distinct with approx_count_distinct.
- Fix joins next: explicitly broadcast the small side (F.broadcast) when it truly fits in executor memory, fall back to sort-merge for large-on-large, and for severe skew apply salting (random suffix on the hot key, exploded on the other side).
- Right-size partitions to 128-256MB each and switch on AQE, so partition counts and skewed joins keep adjusting at runtime instead of being frozen at plan time.
- Cache only DataFrames reused across multiple actions, materialize with a count, unpersist when done; never collect large data to the driver, take(n) exists for a reason.
- Verify with explain(mode="cost") and a partition skew re-check that the new plan actually removed the extra shuffle stages before calling the job tuned.
One power source. 6 lines out.
spark-optimization · core
core active · 6 lines
- ✓ speed up slow spark jobs
Speed up slow Spark jobs and ETL pipelines
- ✓ diagnose data skew domin…
Diagnose data skew dominating job runtime
- ✓ right-size partitions to
Right-size partitions to 128-256MB
- ✓ choose broadcast vs sort…
Choose broadcast vs sort-merge vs bucket joins
- ✓ tune executor memory to
Tune executor memory to stop OOM and spills
- ✓ read explain plans to find
Read EXPLAIN plans to find full scans
Yours to keep.
Drag time forward. Watch what stays.
Forever
That's what owning means.
The rented stack
ai writing tool: subscription
expired · access lostanalytics suite: subscription
expired · access lostdesign platform: subscription
expired · access lost(nothing left)
Your forge
-
Cut runtime by minimizing the most expensive operation: shuffle
license: perpetual -
Lower cluster spend with auto-scaling and right-sizing
license: perpetual -
Stop one skewed partition from holding up the whole job
license: perpetual -
Read 10-100x less I/O with columnar formats and pushdown
license: perpetual
subscriptions expire · deeds don't
Everything in the box.
Pick a piece up. Watch it work.
AQE-enabled optimized SparkSession config template
6 parts · one working system · ships instantly by email
This wasn't forged for everyone.
- Not for you if you'd rather rent a tool than own one.
- Not for you if you want someone else to run your stack.
- Not for you if you're happy guessing.
Data engineers running Spark pipelines who need slow jobs to run fast, scale to large datasets, and stay within cluster budget.
then this was forged for you.Works with
Universal by design: these run in any AI. Delivered in the open Agent Skills + MCP format (native in Claude); ChatGPT, Gemini, Cursor and Copilot adapt the same files their own way.
- Claude Native format
- ChatGPT Adapts via open standards
- Gemini Adapts via open standards
- Cursor Adapts via open standards
- Copilot Adapts via open standards
Catch what's on your mind.
the air is clear. nothing between you and the forge.
catch a spark: the forge will answer
-
Does this apply to managed Spark like Databricks or EMR, or only self-hosted clusters?
The patterns are engine-level, not vendor-level: shuffle minimization, 128-256MB partition sizing, join strategy selection, and executor memory breakdown work wherever Spark runs. The code examples are PySpark, and the AQE-enabled SparkSession config template drops into any environment that lets you set Spark configs.
-
Spark already has Adaptive Query Execution. Why do I need a playbook on top of it?
AQE handles moderate skew and partition coalescing automatically, but it will not pick broadcast vs bucket joins for you, salt a severely skewed key, or explain why a stage spills to disk. The playbook covers the decisions AQE cannot make, including manual salting and reading EXPLAIN plans to find full scans.
-
Will it auto-tune my cluster or fix jobs without my involvement?
No. It is a set of patterns, a config template, and skew-detection monitoring snippets, not an agent that rewrites your pipelines. You still read your own stage metrics, identify the bottleneck, and apply the matching pattern yourself.
-
How is it delivered?
By email right after purchase: ready to run, downloaded instantly, no setup wait.
-
One-time or subscription?
A one-time purchase; no subscription or hidden fees. VAT (20%) is included.
-
Can I get a refund?
As a digital product, it can’t be refunded once downloaded. That’s why we show exactly what’s inside and who it’s for, right here.