Serverless is a contract with an allow-list

The €100 Lakehouse — 6/12

Databricks serverless refused three of my classic idioms. The best error had an empty string where the type name should be — at table 21 of 43, with twenty Delta targets already created.

Serverless is the deliberate choice behind the €100/mo ceiling (auto_stop_mins=1). It is also a contract with an allow-list — and I paid for each clause in hours.

→ spark._jvm, the normal PySpark way to touch the FileSystem: refused with [JVM_ATTRIBUTE_NOT_SUPPORTED]. The error's suggested fix is a single-user cluster — pay more to keep a private idiom. The actual fix: dbutils.fs.

→ Postgres has a `time` type. Spark doesn't. The parquet carried TIME(MICROS) and the job died with "data type is not supported" — type name blank. Fix: cast to TEXT in the extraction SELECT; '09:30:00' sorts and reads fine.

→ psycopg[binary]>=3.2 died with SIGABRT (exit 134) inside the C loader. The serverless image ships its own psycopg; the version range accepted it, and only psycopg-binary was installed fresh — pure-Python package and compiled shim from different releases. Fix: pin both to 3.3.4, same as uv.lock.

Cheap serverless is a contract with an allow-list — every refused idiom has a supported equivalent, and "upgrade a tier" is almost always the wrong answer to a price.

What's the most expensive error message you've ever debugged — and did it even name the thing that was broken?

Databricks · dbt · Airflow · Terraform · AWS

P.S. New tech post every Wednesday.

#100EuroLakehouse #Databricks #Serverless