The Challenge
The CBAS (Community-Based Accountable Care) program data was processed by a collection of standalone Python scripts — each one written at a different time, by different people, with different assumptions about data formats, error handling, and scheduling. The scripts worked, but they were fragile: a schema change in an upstream file could silently break a pipeline, and there was no centralized monitoring or alerting. When something failed, the team often found out from a stakeholder asking why a report was late.
The Brie AML (Anti-Money Laundering) data had a similar problem but with higher stakes. AML data flows through an ingestion layer into a lakehouse, and the two layers must agree — any discrepancy could mean a compliance gap. The existing reconciliation was manual and infrequent, leaving the team exposed to undetected data quality issues.
The Approach
A unified Databricks pipeline architecture replaced the scattered Python scripts. Each CBAS program — SCAL, NCAL, and FQHC — got its own pipeline, but all three shared a common framework: standardized ingestion with schema validation, a transformation layer with business rule enforcement, and an output layer with automated quality checks. Databricks Workflows orchestrate the pipelines, providing centralized scheduling, monitoring, and alerting.
For the Brie AML reconciliation, a SQL-based validation framework compares the ingestion and lakehouse layers record by record. The framework checks not just row counts but field-level consistency — amounts, dates, statuses, and identifiers — and generates a detailed mismatch report whenever discrepancies surface. The reconciliation runs automatically as part of the daily pipeline, so compliance issues surface within hours rather than at the next manual review.
On the reporting side, Power BI dashboards give program managers a self-service view of membership data and CBAS metrics. The dashboards connect directly to the lakehouse, so they always reflect the latest data without manual refreshes or exports.
The Outcome
The consolidated Databricks pipelines eliminated the fragility of the old script-based approach. Schema changes are now caught at ingestion with clear error messages, and pipeline failures trigger automatic alerts with enough context to diagnose the issue quickly. The team went from reactive firefighting to proactive monitoring — pipeline health is visible at a glance, and issues are caught before they affect downstream reports.
The automated Brie AML reconciliation reduced the compliance risk window from weeks to hours. The Power BI dashboards gave program managers direct access to the data they needed, eliminating the backlog of ad-hoc data requests that had previously consumed significant analyst time. Membership summaries and CBAS metrics that once required days of manual work are now available on demand.
Key Learnings
The key insight was that data pipeline reliability is not about writing perfect code — it is about building systems that fail loudly and recover gracefully. By adding schema validation at ingestion, automated quality checks at every layer, and centralized monitoring, the pipelines became self-diagnosing. The second lesson was the value of self-service reporting: when program managers can answer their own questions from a dashboard, the data team is freed to focus on building better pipelines rather than fulfilling report requests.