DuckDB and Apache Iceberg on AWS Glue: the SQL that works and the limits

Thunderduck runs every query in its own DuckDB worker, and since August our managed Iceberg catalogs live in AWS Glue. Reading from Glue was easy. Writing to it took longer than expected, because three separate systems each did something reasonable that added up to a failure. This post covers the setup that works, why the obvious CREATE TABLE fails, and what still does not work. All the SQL is plain open-source DuckDB 1.5, so you can run it on a laptop with AWS credentials.

Attaching Glue takes four statements

INSTALL iceberg; LOAD iceberg;
INSTALL aws;     LOAD aws;

CREATE SECRET glue_creds (TYPE S3, PROVIDER credential_chain, REGION 'us-east-1');

ATTACH ':' AS glue (TYPE ICEBERG, ENDPOINT_TYPE 'GLUE');

• ':' means the default Glue catalog of whatever account the credentials belong to, so the same SQL works unchanged in every environment.
• credential_chain picks up the role the process already has: an ECS task role, an EKS service account or a local profile. No keys appear in SQL.
• The secret has to exist before the ATTACH. Otherwise you get "Could not find a valid storage secret (s3 or aws)".
• Install the aws extension up front. DuckDB will autoload it, but that is a download in the middle of a query. We bake it into our images for that reason.

After that, each Glue database is a schema under glue, and reads work as you would hope. DuckDB uses Iceberg partition values and per-file min/max statistics to skip files, so a filter on the column the table is partitioned or sorted by reads a fraction of the data.

SELECT order_date, count(*) AS orders
FROM   glue.analytics.orders
WHERE  order_date >= DATE '2026-09-01'
GROUP  BY 1;

SELECT * FROM iceberg_snapshots('glue.analytics.orders');  -- commit history
SELECT * FROM iceberg_metadata('glue.analytics.orders');   -- manifests and files

Why CREATE TABLE fails, in three layers

The obvious CREATE TABLE ... AS SELECT into glue.analytics returns a 400 from Glue: "Location information cannot be null while creating an iceberg table". We found three causes before it worked.

• Glue's Iceberg REST endpoint requires a storage location on every table create.
• DuckDB's iceberg extension does not send one (duckdb-iceberg issue #549).
• Glue does not fall back to the database's LocationUri. We set one on the database and got exactly the same 400, which ruled out the fix we had planned.

DuckDB 1.5 has a way to pass the location: a WITH clause between the table name and AS.

CREATE TABLE glue.analytics.orders_daily
    WITH ('location' = 's3://my-lake/analytics/orders_daily/')
AS
SELECT date_trunc('day', created_at) AS day, count(*) AS orders
FROM   glue.analytics.orders
GROUP  BY 1;

There was a third layer specific to us. Thunderduck parses and rewrites every statement before it runs, to resolve catalog names and enforce tenant boundaries. Our SQL parser silently dropped that WITH property when it regenerated the statement, so a user who typed the right clause still got the same Glue 400. We now derive each table's location from the catalog's own prefix and append the clause to the statement text after rewriting. A location the user types is accepted only if it falls inside that prefix, because in a shared bucket a free-form location would let one tenant write into another's files.

The IAM a DuckDB process needs

• Read: glue:GetCatalog(s), glue:GetDatabase(s), glue:GetTable(s), glue:GetPartition(s), plus s3:GetObject and s3:ListBucket on the table locations.
• Write: add glue:CreateTable and glue:UpdateTable, plus s3:PutObject on the table's prefix.
• Check a role before deploying with aws iam simulate-principal-policy. It reports implicitDeny for each missing action.
• Our planning service only ever has read access. We run EXPLAIN on every query before executing it, and a CREATE TABLE plan never calls Glue's create API. Keeping that role read-only means planning a query can never create a table as a side effect.

What still does not work

• DROP TABLE fails through the Glue endpoint, even for an admin principal. Drop with aws glue delete-table, then aws s3 rm --recursive on the table's prefix.
• SHOW TABLES on a database that does not exist returns zero rows, not an error. That is how a typo looks like an empty schema.
• There is no information_schema on an Iceberg catalog. Use SHOW TABLES and DESCRIBE.
• Glue database names allow only lowercase letters, digits and underscores.
• Every write is a new snapshot. Frequent small writes need regular compaction and snapshot expiry, or reads slow down as the number of files grows.

On Thunderduck

All of the above is set up for you. You create an Iceberg-on-Glue catalog by giving it a name. The Glue database, its storage prefix, the secret, the ATTACH and the table locations are handled for you, so CREATE TABLE ... AS SELECT works with no WITH clause. Compaction, sorting and snapshot expiry run from the LakeOps page. The step-by-step reference, with every statement and limit above, is in the docs: https://docs.thunderduck.ai/guides/duckdb-iceberg-glue/

CREATE TABLE lake.default.orders_daily AS
SELECT date_trunc('day', created_at) AS day, count(*) AS orders
FROM   sales_pg.public.orders
GROUP  BY 1;