A proof that could have failed

The €100 Lakehouse — 5/12

The equivalence check passed: fingerprint 35324916267, identical on both paths. Then I deleted the baseline and ran it again.

The decision: retire a Phase-0 shortcut where dlt wrote Databricks directly. Now dlt ends at an S3 prefix, and bronze is written by exactly one thing — Auto Loader, incremental batch, trigger=availableNow.

The proof:

→ fingerprint = sum(crc32(concat_ws('~', 18 columns))), equal on both paths
→ 17 = 17 rows, bidirectional EXCEPT diff: 0 rows
→ re-run: batches=0 — idempotency measured, not assumed

But MERGE-ing 17 identical rows over 17 identical rows returns 17 whether the new landing works or not. My check was tautological until I removed the target. So I refused the green result, dropped the baseline table, and made the pipeline rebuild it from nothing. Same fingerprint. Now it counts.

The honest cost is written in the spec: a load now has TWO commit points — the S3 write and the stream checkpoint commit — and all the interesting failures live in the interval between them. Hence the rule: the checkpoint and the bronze table reset TOGETHER. Always. Never one without the other. One composed act in code, no path that does half.

An equivalence proof only counts if it could have failed.

When did you last delete a passing test's baseline just to check the test could still fail?

Databricks · dbt · Airflow · Terraform · AWS

P.S. New tech post every Wednesday.

#100EuroLakehouse #Databricks #DataEngineering