Data & Analytics Skill
Analytics Tracking
Set up, improve, or audit analytics tracking and measurement.
$15
Airflow DAG Patterns
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and…
A production playbook for building Apache Airflow DAGs the right way, with battle-tested patterns for operators, sensors, branching, testing and deployment. It centers on the principles that keep pipelines reliable: idempotent, atomic, incremental and observable tasks, and shows how to apply them with the modern TaskFlow API. Every pattern comes as runnable code you can adapt rather than reinvent.
$15 one-time
Add to a kit → Prices include 20% VAT. · Forged on real agency work · one-time, no lock-in
- Type Skill
- Category Data & Analytics
- Delivery Email · instant
- License One-time
Inside the run · no black box
See the actual work before you buy it.
A DAG that cannot survive a backfill is not production-grade. Every pipeline this skill builds is idempotent first, observable second, and only then scheduled:
- Designs every task around the execution date macro instead of datetime.now(), so retries and backfills always produce the same result; writes are UPSERT or temp-plus-atomic-rename, never blind INSERT, and depends_on_past stays off so one bad day never locks the whole backfill.
- Builds the pipeline with the TaskFlow API: each ETL step is a @task function whose return value passes through XCom automatically, heavy logic stays in imported modules so the DAG file remains pure orchestration, and large payloads go to S3 with only the path passed through XCom.
- Sets every sensor to reschedule mode with an explicit timeout and a poke interval matched to the source, so waiting for an S3 file, an external DAG or an API never occupies a worker slot for hours.
- Wires failure, retry and SLA-miss callbacks that ship dag_id, task_id, execution date, the exception and the log URL to the alert channel; cleanup tasks run on the ALL_DONE trigger rule even when upstream fails, so nothing is left half-done silently.
- Tests the DagBag in CI before anything deploys: zero import errors, no dependency cycles, expected task count and schedule verified, plus plain unit tests on the extract and transform functions themselves.
- Scales repeated pipelines through a create_dag(config) factory that reads YAML or Airflow Variables, gives each generated DAG a unique id and tags, and watches scheduler parse time as the DAG count grows so 500 configs never melt the scheduler.
One power source. 6 lines out.
airflow-dag-patterns · core
core active · 6 lines
- ✓ build an etl pipeline with
Build an ETL pipeline with clean TaskFlow API tasks and automatic XCom passing
- ✓ generate many similar dags
Generate many similar DAGs from config with a factory pattern
- ✓ add branching and condit…
Add branching and conditional logic driven by data-quality checks
- ✓ wait on external files
Wait on external files, S3 keys or upstream DAGs with reschedule-mode sensors
- ✓ wire failure, retry and
Wire failure, retry and cleanup callbacks for proactive alerting
- ✓ unit-test dag structure
Unit-test DAG structure, dependencies and cycle-freedom in CI
Yours to keep.
Drag time forward. Watch what stays.
Forever
That's what owning means.
The rented stack
ai writing tool: subscription
expired · access lostanalytics suite: subscription
expired · access lostdesign platform: subscription
expired · access lost(nothing left)
Your forge
-
Ship pipelines that are safe to retry and backfill thanks to idempotent design
license: perpetual -
Free up worker slots and cut cost with reschedule-mode sensors and timeouts
license: perpetual -
Catch silent failures early with callback-driven Slack/PagerDuty observability
license: perpetual -
Scale to many pipelines without scheduler slowdown using dynamic DAG generation
license: perpetual
subscriptions expire · deeds don't
Everything in the box.
Pick a piece up. Watch it work.
TaskFlow API ETL pattern with automatic XCom and modular import discipline
6 parts · one working system · ships instantly by email
This wasn't forged for everyone.
- Not for you if you'd rather rent a tool than own one.
- Not for you if you want someone else to run your stack.
- Not for you if you're happy guessing.
Data engineers building or hardening Apache Airflow pipelines who want production-grade, idempotent, well-tested DAG patterns.
then this was forged for you.Works with
Universal by design: these run in any AI. Delivered in the open Agent Skills + MCP format (native in Claude); ChatGPT, Gemini, Cursor and Copilot adapt the same files their own way.
- Claude Native format
- ChatGPT Adapts via open standards
- Gemini Adapts via open standards
- Cursor Adapts via open standards
- Copilot Adapts via open standards
Catch what's on your mind.
the air is clear. nothing between you and the forge.
catch a spark: the forge will answer
-
Does it assume a specific Airflow version or hosting like MWAA or Composer?
The patterns are built around the TaskFlow API and standard operators, so they apply on managed Airflow as well as self-hosted. They are DAG-authoring patterns, not tied to one host.
-
My DAGs already rerun on retry, so why push idempotency so hard?
A DAG that reruns is not the same as one that produces the same result when it reruns, and that gap is where silent data duplication hides. Idempotent and atomic tasks are what make a retry safe rather than just possible.
-
Does it provision the Airflow cluster too?
No, it covers how to author reliable DAGs, not how to stand up or scale the infrastructure. Deploying and operating the Airflow environment is separate.
-
How is it delivered?
By email right after purchase: ready to run, downloaded instantly, no setup wait.
-
One-time or subscription?
A one-time purchase; no subscription or hidden fees. VAT (20%) is included.
-
Can I get a refund?
As a digital product, it can’t be refunded once downloaded. That’s why we show exactly what’s inside and who it’s for, right here.