Zero ETL first — then ETL, when you actually need it

Most data teams start a project by deciding where to move the data. Bucket to warehouse, database to bucket, staging to mart. It feels like the first step because it always has been. It isn't — it's the step you take when the engine can't reach the data, and ours can.

So the honest order of operations is: query it where it lives first, and move it only when the workload actually pays for the pipeline. Both halves are supported natively. Here's how to tell which half you're in.

Federated queries: the join happens, the copy doesn't

A federated query joins Iceberg and Parquet in S3, GCS or Azure with live PostgreSQL and MySQL, in memory, in one statement. Nothing is staged, nothing is duplicated, and nothing has to finish loading before you can ask the question. What you get for that:

• No pipeline to build, schedule, monitor or repair. Most teams wrote half their ETL purely to park operational data next to lake data so something could join them. That half can be deleted.
• No sync lag. You read the row the OLTP replica has right now — not the one last night's load happened to catch. Freshness bugs stop being a category.
• No second copy to secure, budget for or forget about. Data stays in your buckets in open formats, so the exit cost of Thunderduck is a deleted credential.
• Days, not quarters. Exchange credentials in the console, let the engine discover catalogs, schemas, logical sizes and index usage, and start querying — no modeling project stands between you and the first answer.
• The same catalog for people and agents. Analysts get SQL; agents get one MCP endpoint over the identical catalog, with the same guardrails.
• A cost estimate before execution, not an invoice after it. Federation without a pre-flight check is how a curious agent full-scans 400 GB. Every query — human or agent — gets an estimated cost and runtime first, and anything over budget is blocked or rerouted.

Zero ETL is a default, not a religion

Federation answers most questions, but not all of them. There are workloads where materializing a table is simply the cheaper, kinder engineering decision, and pretending otherwise would be marketing rather than architecture. You want a physical table when:

• The same expensive join runs hundreds of times a day. Compute it once at 03:00 instead of paying for it on every dashboard load.
• The source is a production replica you would rather touch once than continuously, however good your concurrency caps are.
• You need history the source doesn't keep — slowly changing dimensions, point-in-time snapshots, an audit trail of what a record looked like last March.
• Data arrives incrementally and late, and you want a stable, deduplicated table rather than a window function re-deriving the truth on every read.
• The raw files are small and fragmented, and compaction into well-sized Parquet or Iceberg files is worth real money on every subsequent scan.

ETL, on the same engine you already query with

When you decide to materialize, there is nothing new to buy or deploy. It's SQL, on the same engine, writing into the same catalogs you already read from — so a table built by a load is indistinguishable from any other table in Studio, in the JDBC driver and over MCP.

CREATE TABLE analytics.orders_enriched AS
SELECT
  o.order_id,
  o.customer_id,
  c.segment,
  o.order_ts,
  o.amount
FROM lake.raw.orders             AS o   -- Parquet in S3
JOIN pg_replica.public.customers AS c   -- live PostgreSQL replica
  ON c.id = o.customer_id
WHERE o.order_ts >= DATE '2026-01-01';

Incremental loads are the same idea with a watermark: read only what arrived since the last run, append it, and let a scheduled job fire the statement on a cron expression in your own timezone. Runs never overlap themselves, and each one lands in query history with its plan, timings and cost, exactly like an interactive query.

INSERT INTO analytics.orders_enriched
SELECT
  o.order_id,
  o.customer_id,
  c.segment,
  o.order_ts,
  o.amount
FROM lake.raw.orders             AS o
JOIN pg_replica.public.customers AS c
  ON c.id = o.customer_id
WHERE o.order_ts > (
  SELECT COALESCE(MAX(order_ts), TIMESTAMP '1970-01-01')
  FROM analytics.orders_enriched
);

From source to mart: Medallion with dbt

One CREATE TABLE AS is a load. A dozen of them, depending on each other, is a platform you now maintain by hand. That's the point where dbt earns its place, and where the Medallion layout stops being a diagram and starts being useful:

• Bronze — raw, as landed. Often not materialized at all: with federation, bronze can simply be a view over the files and tables that already exist.
• Silver — cleaned, typed, deduplicated, conformed keys. This is where most of the incremental logic lives, and where tests earn their keep.
• Gold — marts shaped for a question: one table per business process, aggregated and joined so a dashboard or an agent doesn't have to re-derive anything.

Point Thunderduck at a dbt project on GitHub — repo URL, branch, an optional subdirectory for monorepos, and a token that goes straight to Secrets Manager and never appears in a query or a log. Refresh syncs the project to a commit; Run executes build, run, test, seed, snapshot or compile with your own --select and --full-refresh flags. Models materialize as catalog tables you can query the moment the run finishes, and every run keeps its commit SHA, logs and artifacts. You edit models in your own IDE and push — there is no separate orchestration product in the middle.

Because dbt runs on the same engine as everything else, each model's build shows up in query history with its plan and its cost. Which means the question “what does our gold layer cost us per night, model by model?” has an answer, in the same place as every other query.

The semantic layer your catalog — and your agents — have been missing

The most under-rated part of dbt isn't the DAG, it's the definitions around it. Descriptions, tests, and semantic models with their entities, dimensions and metrics turn a warehouse of columns into something a newcomer — or a language model — can actually read.

• One definition of “active customer”, written once in a repo, reviewed in a pull request, instead of six subtly different definitions living in six dashboards.
• A catalog that describes the business, not just the storage. Table and column descriptions ship with the models, so discovery answers “what does this mean” and not only “what type is this”.
• Agents that ask for a metric instead of inventing a join. Given a defined metric and its dimensions over MCP, an agent composes the query it was designed to compose — far less room to hallucinate a plausible, wrong number.
• Cheaper agent exploration. An agent pointed at a modeled gold mart scans megabytes; the same question against raw bronze scans hundreds of gigabytes. The semantic layer is a cost control as much as a clarity one.
• Governance that already exists. Metric changes are code review, version history and tests — the workflow your engineers use for everything else.

A semantic layer is the difference between an agent that can query your data and an agent that understands it.

Today the definitions live in your dbt project and the models land in your catalogs, queryable by analysts and agents alike. Pulling that semantic metadata deeper into catalog discovery — so an MCP client sees metrics and dimensions, not just tables and columns — is what we're building next. If you're already modeling in dbt, we'd like to build it against your project.

The rule of thumb

• Start federated. Zero pipelines, zero lag, and you learn what the data actually looks like before you commit a schema to it.
• Materialize when repetition, replica load, history or file layout makes the copy cheaper than the scan — with CREATE TABLE AS and a scheduled incremental load.
• Model with dbt when more than one team depends on the answer, and let the semantic definitions feed your catalog and your agents.

Zero ETL isn't a promise that you'll never write a pipeline. It's a promise that you'll only write the ones that earn it — and that the engine won't charge you a migration to find out which.