Airflow EKS to MWAA Migration: A Data Engineer's Playbook for Reconciliation DAGs That Scale

Banking reconciliation DAGs across multiple Systems of Record (SORs) outgrow self-managed Airflow fast. Running daily reconciliation across banking SORs surfaces every pain point of Airflow on EKS — version upgrades, worker scaling, scheduler tuning, and monitoring. Here is the EKS to MWAA migration playbook that fixed it.

Why MWAA won over self-managed EKS

Running Airflow on EKS gives you full control, but control is expensive. Every Airflow version upgrade meant testing Helm charts, validating provider packages, and scheduling downtime. Worker autoscaling needed constant tuning. Monitoring the scheduler, webserver, and metadata database was a second job on top of building data pipelines.

Amazon MWAA (Managed Workflows for Apache Airflow) takes that operational burden away. Upgrades become a version bump, scaling is automatic, and CloudWatch integration gives you scheduler metrics out of the box. For a data engineering team whose real job is reconciliation logic — not Kubernetes maintenance — that trade is worth making.

What changed in the DAGs

• MWAA for managed ops: upgrades, scaling, and monitoring without EKS toil.
• Modular task groups + reusable operators: one pattern per SOR, driven by YAML.
• YAML-driven configs: regional params, SOR metadata, job settings — same DAG, many deployments.
• Observability: tagging standards, SLAs, email alerts on mismatch.

The biggest code change was standardizing on task groups. Each SOR pair gets the same five-step pattern — extract, validate, compare, report, alert — implemented once as a reusable task group and parameterized per SOR. Adding a new SOR pair is now a YAML change, not a development project. The full framework is documented in the banking reconciliation project.

YAML-driven configuration

sor_pair:
  source: core_banking
  target: payment_gateway
  schedule: "0 6 * * *"
  thresholds:
    amount_tolerance: 0.01
    max_missing_pct: 0.001
  alerts:
    - data-engineering@company.com
    - reconciliation-owners@company.com

One DAG file reads configs like this and generates a task group per SOR pair. Regional variations — different schemas, different schedules, different alert routing — all live in YAML. The DAG code never changes when a new region onboards.

Regression must-dos

Run old vs new DAGs side-by-side in QA, diff reports row-for-row, and gate Prod promotion on parity. SQL validation between ingestion, processing, and consumption layers in Databricks adds the safety net — catching issues at the ingestion layer is always cheaper than discovering them in final reports.

The data engineer’s checklist

1. Freeze a golden dataset from the EKS DAGs before migration.
2. Run EKS and MWAA DAGs in parallel in QA for at least two full cycles.
3. Diff every report row-for-row, not just row counts.
4. Gate production promotion on 100% parity.
5. Set SLAs and alerts on day one — silent DAGs are broken DAGs.

Migrating a data platform? Platform migrations are the day job — get in touch. More on the validation layer in the Databricks reconciliation notes.