Your whole pipeline, nothing to host.
Your DBT project, its schedule, its alerts and the upkeep of your Iceberg tables, on the same engine that runs your queries. No Airflow to host, no Kubernetes to patch, no separate orchestration bill.
The warehouse is only half the work.
Around every warehouse sits a second platform: something to run DBT, something to schedule it, something to watch it, and scripts to keep the tables healthy. Each one is another thing your team maintains instead of building.
- A warehouse to store and query the data
- A paid DBT service, or Airflow you host yourself on Kubernetes
- A monitoring tool to find out a model went stale, and another to see who used what
- Hand-written Spark jobs to compact and clean up Iceberg tables
- One engine for queries, DBT runs and scheduled jobs
- Pipelines defined next to your models, run straight from Git
- Freshness, alerts, and every run's CPU and memory by user, job or agent
- LakeOps: compaction and snapshot cleanup from one page
Pipelines built in, not bolted on.
Add a thunderduck.yml next to your dbt_project.yml and it decides what runs when: which models, in what order, on which schedule and timezone. Production runs whatever is merged to your default branch. Each run is sized like any other query and lands in the same history.
- Run, test, seed and snapshot on the same engine as everything else
- Freshness for every model, tracked against its schedule
- An alert when a run fails or a model goes stale
Try the change on real data. Merge when it's right.
A branch is a test environment
Run any Git branch and it builds into dev catalogs, in a schema named after the branch, reading production data. Production never changes until you merge.
Isolated compute per run
Every DBT run gets its own engine, sized from its plan. A heavy backfill on a branch can't slow down production or your colleagues.
Agents ship models too
Over MCP an agent pushes a branch, runs it, checks the tables and opens the pull request. You review and merge.
Iceberg upkeep, without the monster.
Iceberg tables need constant care: streams and incremental loads leave thousands of small files, and every write adds a snapshot that has to be expired one day. Most open-source platforms leave that to you and a cluster of Spark jobs. LakeOps shows which tables need attention and fixes them from one page.
- Rust-based compaction rewrites small files into 128, 256 or 512 MB ones
- Spark-based snapshot expiry and orphan-file cleanup
- Each operation commits atomically, so queries keep reading while it runs
- Health rules only flag a table when the operation would actually change it
Who ran what, and what it cost.
Why is the bill up? Which job is hogging memory? Was that a person, a schedule or an agent? Query history answers the questions data engineers get every day: every run from Studio, the API, MCP agents, scheduled jobs and DBT, with the CPU and memory it actually used.
- Filter by user, source, API token, scheduled job, DBT repo, status and time range
- Sort by CPU, peak memory or run time to find the heaviest runs
- People, systems and AI agents each show up as their own source
- Open any run to see its SQL, plan and timings, or replay it in Studio
DLT workloads that agents set up for you.
In an upcoming release, agents will build and run DLT pipelines that pull from SaaS apps, APIs and databases into your managed Iceberg catalogs, then keep them loading on a schedule.
Pipelines are new. Help us shape them.
Tell us how your team runs DBT today and what your orchestration has to do. We work with early teams directly, and what you need shapes what we ship next.
