Every team running both dbt and Flink has had the same conversation at some point: why are we maintaining two completely separate transformation stacks? SQL here, SQL there, but different CI, different testing, different deployment rituals. Confluent finally shipped a real answer in late March — dbt-confluent, an adapter that lets you dbt run directly against Flink compute pools. Xebia's open-source dbt-flink-adapter has been around longer, but the Confluent release brings enterprise backing and a smoother onramp for teams already on Confluent Cloud.

I spent the past two weeks poking at both. The good news: this genuinely works. The bad news: Flink's session semantics and stateful job management introduce failure modes that batch-only dbt users have never had to think about.

How It Actually Works

The dbt-confluent adapter talks to Flink through the SQL Gateway REST API. You write standard SQL SELECT statements as dbt model files. dbt run compiles them and submits Flink SQL jobs to your compute pool. Three materializations ship today:

  • streaming_source — declares a Kafka topic as a dbt source

  • view — creates a virtual Flink SQL view over streaming data

  • streaming_table — a continuously updating result set, and the one you actually want for most use cases

Your profiles.yml looks like this:

my_streaming_project:
  target: dev
  outputs:
    dev:
      type: confluent
      organization_id: "{{ env_var('CONFLUENT_ORG_ID') }}"
      environment_id: "{{ env_var('CONFLUENT_ENV_ID') }}"
      compute_pool_id: "{{ env_var('CONFLUENT_POOL_ID') }}"
      flink_api_key: "{{ env_var('CONFLUENT_FLINK_API_KEY') }}"
      flink_api_secret: "{{ env_var('CONFLUENT_FLINK_API_SECRET') }}"
      dbname: my-kafka-cluster
      threads: 1

From there, the workflow feels familiar: models/, tests/, dbt build. The adapter handles Flink SQL statement lifecycle behind the scenes. If your team knows dbt, the learning curve is the streaming semantics, not the tooling.

The 10-Minute Session Time Bomb

Here's where things get spicy. Flink's SQL Gateway operates in sessions, and those sessions expire after 10 minutes by default. Tables and views created in one session are invisible to another.

Run dbt run, wait 11 minutes, run dbt test — everything fails. Flink can't find your objects. The only recovery is re-running the entire model, which for streaming jobs means reprocessing from your last checkpoint or, worse, from the beginning of the topic.

The Confluent adapter handles this more gracefully than the open-source Xebia version since it manages sessions server-side, but you still need to internalize that streaming dbt is not "run once and forget." These are long-lived jobs. Your mental model has to shift from "transformation that executes and completes" to "deployment that starts and keeps running." That's a bigger cognitive shift than it sounds.

Stateful Upgrades: Where It Gets Ugly

Batch dbt is stateless by nature. You drop and recreate tables, no harm done. Streaming Flink jobs accumulate state — windows, aggregations, joins all build up operator state that vanishes if you naively restart.

The dbt-flink-adapter tries to handle this through a savepoint workflow:

materialized: table
upgrade_mode: savepoint
job_state: running
execution_config:
  pipeline.name: my_aggregation_job
  parallelism.default: 4
  state.savepoints.dir: s3://my-bucket/savepoints/

When you redeploy, the adapter stops the running job with a savepoint, drops and recreates the Flink SQL structures, then restarts from that saved state. In theory.

In practice, this is fragile in ways that matter. If deployment fails between the stop and restart steps, you're left with a stopped job and a savepoint you need to manually recover. The adapter is stateless itself — it doesn't track what happened between runs. No automatic recovery if your session cluster has problems either.

And the real kicker: Flink SQL doesn't let you set operator UIDs. If your SQL changes touch a stateful operator — say you modify a windowed aggregation — the savepoint restore can just fail because Flink can't map the old state to the new operator graph. You'll find yourself dropping state and reprocessing from scratch anyway. For a two-hour window that's annoying. For a seven-day window over a high-volume topic, that's a very expensive afternoon.

Who Should Adopt This Today

The honest answer depends on which adapter and what kind of jobs you're running.

dbt-confluent on Confluent Cloud is the safer bet if you're already in that ecosystem. Managed infrastructure absorbs a lot of the session and state pain. Stateless transformations — filters, enrichments, simple joins without windowing — work well right now. Start there.

dbt-flink-adapter (Xebia) is more flexible but demands real operational maturity. You need to understand Flink's state backend, savepoint mechanics, and SQL Gateway session management before you deploy anything beyond a demo. If your team already runs Flink in production and wants dbt's testing and lineage features layered on top, it's worth evaluating seriously.

Skip both if your streaming jobs are heavily stateful — complex windowed aggregations, large-scale joins with state TTL management — and you need zero-downtime upgrades. The adapter layer adds a failure surface that Flink's native deployment tools handle better. You're trading operational control for developer convenience, and for stateful workloads that trade isn't worth it yet.

The Gaps That Still Hurt

No incremental materializations. No snapshots — Flink SQL lacks the MERGE and UPDATE operations dbt snapshots require. No seeds, though static CSV loading doesn't really make sense in a streaming context anyway. The testing story uses deterministic snapshot queries, which is clever but limited compared to what you get testing batch models against a warehouse.

The Confluent adapter also pins threads: 1, so models deploy sequentially. For large DAGs with dozens of models, this gets slow. Parallel deployment is on the roadmap but hasn't shipped.

Where This Lands

The "one SQL workflow for batch and streaming" pitch is genuinely compelling, and the tooling is further along than I expected. For simple streaming transformations — topic-to-topic enrichments, filtering, lightweight aggregations — this is production-viable today. Confluent clearly invested in making the developer experience feel like regular dbt, and it shows.

But if you hear "just add streaming to your existing dbt project" and picture the same smoothness as adding a new Snowflake model, recalibrate hard. Streaming introduces fundamentally different lifecycle semantics: jobs that run forever, state that matters, sessions that expire, upgrades that can destroy hours of accumulated computation. dbt on Flink works. It just doesn't work the same way.