dbt on Flink Won't Unify Your Data Stack
Three days ago Confluent dropped the dbt-confluent adapter, and the data engineering corner of the internet lost its mind. One tool for batch and streaming. Same ref() graph whether you're hitting Snowflake or Flink. The dream, right?
I've spent the last 72 hours reading the docs, the source, and the GitHub issues. Here's my take: dbt on Flink doesn't eliminate the batch/streaming divide — it just hides it behind familiar syntax until something breaks at 3 AM.
The pitch sounds perfect
pip install dbt-confluent, write SQL models, dbt run. Streaming pipelines next to warehouse transforms, one lineage graph, one CI/CD pipeline. Confluent even nailed the testing story — snapshot queries return bounded, point-in-time results instead of the usual streaming-test nightmare. And the Iceberg integration means Flink writes to S3 while Snowflake or BigQuery reads directly. No connector spaghetti.
So what's the problem?
State management didn't go anywhere
Flink is a stateful stream processor. That's its superpower and its sharpest edge. When you write a streaming_table materialization, you're deploying a long-running Flink job that accumulates state — windows, aggregations, join buffers. That state needs checkpoints, savepoints, and careful lifecycle management.
The dbt-flink adapter's lifecycle handling is lightweight and stateless. If deployment fails between stopping a job with a savepoint and restarting it, you can lose progress. The adapter won't help you if the underlying Flink session cluster has issues. There's no automatic restore from the latest savepoint. You're left staring at the Flink dashboard, manually locating the last good savepoint, and crossing your fingers that the state schema still matches after your model change.
And here's what the docs gloss over: checkpoint storage has real costs. A streaming job maintaining 30 days of session windows can accumulate gigabytes of state. When that checkpoint fails — because the S3 write timed out, because the TaskManager ran out of memory mid-snapshot, because the RocksDB compaction couldn't keep up — your job crashes, and dbt has no opinion about what to do next. In warehouse-land, a failed dbt run is annoying. You re-run it. In streaming-land, a failed deployment can mean data loss, duplicated events downstream, or hours of reprocessing from the last known good offset. The blast radius is fundamentally different, and the adapter doesn't acknowledge that difference.
The session trap
Tables and views created in one Flink session aren't visible in another, and sessions expire after 10 minutes. Run dbt test eleven minutes after dbt run? Everything fails. This isn't a bug — it's how Flink SQL sessions work. But it breaks every assumption dbt users have from years of persistent warehouse catalogs.
What you actually can't do
The adapter doesn't support:
Incremental materializations. Use
streaming_tableinstead, which is conceptually different — it's a continuously running job, not a batch append. If you're porting an existing dbt project that leans on incrementals, you're rewriting those models from the ground up, not just swapping an adapter.Snapshots. Flink SQL doesn't support the transaction primitives dbt snapshots need.
Table renames.
ALTER TABLE ... RENAMEisn't a thing in Flink SQL.Atomic deployments. Operations aren't transactional. A
dbt runthat fails midway leaves you in a partial state with some models updated and others pointing at stale data.
These aren't gaps that'll get patched next quarter. They're fundamental mismatches between what dbt assumes (a warehouse with ACID semantics) and what Flink provides (a distributed stream processor).
Upgrading stateful jobs is still hard
This is the one nobody talks about in the launch posts. When you change a streaming SQL query, you're changing a Flink job's operator graph. Flink needs to map the old state to the new operators, and it can't always do it.
The core issue is operator UIDs. In Java/Scala Flink apps, you explicitly assign UIDs to each operator so the framework knows how to match old state to new topology. Flink SQL doesn't expose this. The planner generates operator IDs automatically, and any change to your query — adding a filter, reordering a join, tweaking a window — can reshuffle those IDs. When the IDs don't match, Flink can't restore from the savepoint, and your only option is to start fresh.
In practice, this means certain model changes require you to drop state and reprocess from scratch. That's fine for a 10-minute tumbling window over a low-volume topic. It's a weekend project for a 30-day sessionization pipeline processing millions of events per hour. And "weekend project" is optimistic — it assumes you have the Kafka retention to replay from, that downstream consumers can handle the backfill spike, and that nobody notices the gap in your real-time dashboard while you're reprocessing. The dbt workflow of "change SQL, run, done" simply doesn't survive contact with stateful streaming reality.
So when does it actually make sense?
Good fit: Mostly stateless transformations — enrichment, filtering, routing. Already on Confluent Cloud. Team knows dbt but not Flink.
Bad fit: Complex stateful pipelines, large windows, session joins, self-managed Flink, or any expectation that dbt run behaves identically across Snowflake and Flink.
The real unification play
The interesting part of this announcement isn't the adapter — it's the Iceberg layer underneath. Flink writes Iceberg, warehouses read Iceberg, and suddenly your streaming output is queryable everywhere without connectors. That's the actual convergence happening in data infrastructure right now. Not "one SQL dialect to rule them all" but "one table format that every engine can read." Iceberg adoption hit 78.6% exclusive usage in recent surveys, and that number will keep climbing.
dbt on Flink is a useful tool for a specific niche. But if someone tells you it unifies batch and streaming, ask them what happens when they need to change a stateful query in production. The answer will tell you everything.