The knob that reads like a connection limit isn't one

The €100 Lakehouse — 4/12

First full load, 42 tables, and the pool blew: "QueuePool limit of size 5 overflow 0 reached". The four pilot tables had fit under five connections by accident.

The mechanism: dlt opens one pipe per resource, and each suspended generator holds a live connection. The default round_robin mode cycles through all of them — so all of them stay open at once.

The obvious fix is max_parallel_items. It reads like a connection limit. I measured it instead of trusting the name:

→ fifo, max_parallel_items=4 → peak: 1 connection. OK.
→ round_robin, max_parallel_items=1 → peak: 5. Failed.

The knob limits items in flight, not open pipes. Here, it controls nothing.

Note which side broke. max_overflow=0 means extra connections get refused — so the promise to the shared production RDS (which also serves the platform API, the legacy ETL and a live Metabase) held, and my load failed. That is the correct side. The alternative exports the problem to the platform, where the symptom is a member who can't book a class.

The measured peaks now live in the settings file and in a test, because the plausible "simplification" reproduces the outage exactly. The load itself: 134,628 rows across 37 tables, first full offline run.

A limit that makes your load fail first is a limit that's working.

What's the best-named knob you've ever met that turned out to control nothing you cared about?

Databricks · dbt · Airflow · Terraform · AWS

P.S. New tech post every Wednesday.

#100EuroLakehouse #DataEngineering #PostgreSQL