Test the idea today, not after the pipeline ships.
Data science is test after test after test. Thunderduck lets you join the dataset you found this morning with your real production data, straight from Jupyter or Colab — no pipeline, no ticket, no waiting.
Most ideas die waiting for a pipeline.
The data that answers a new question is often not in the warehouse yet. Nobody wants to invest in a pipeline before knowing the data is any good — and you can't know until you've tested it.
- 1You find an external dataset that might explain the churn spike.
- 2It isn't in the warehouse, so you ask data engineering for a pipeline.
- 3They have to decide whether it's worth building before anyone knows if the data is any good.
- 4Weeks later it lands — and the idea has gone stale.
- 1Drop the file in your bucket: CSV, Parquet, JSON or Excel.
- 2Join it with your production tables in one query, from your notebook.
- 3Test it as many times as you like, on real data, not a sample.
- 4Only when it proves itself does it become a table or a pipeline.
Test the data you just found against the data you already have.
A file in your bucket is queryable the moment it lands. Read it with read_csv, read_parquet or read_xlsx and join it with live PostgreSQL, MySQL, BigQuery or Iceberg tables in the same SELECT. Nothing is copied, so trying an idea costs a query, not a project.
- CSV, Parquet, JSON and Excel straight from S3, GCS, Azure or R2
- Joined with your databases and warehouse in one query
- Every query priced and checked before it runs, so a wild test can't run up a bill
Your notebook, your frames.
Keep the tools you already use. thunderduck-sqlalchemy is on PyPI, with pandas and Polars support built in: pip install, paste one connection URL with your API token, and read_sql returns a DataFrame. The heavy scan and join run on Thunderduck; only the result comes back to your notebook.
- pip install "thunderduck-sqlalchemy[pandas,polars]"
- Works in Jupyter, Google Colab, VS Code and anywhere Python runs
- Scikit-learn, PyTorch and the rest of your stack read the DataFrame as usual
Promote what proved itself. Drop the rest.
Keep it as a table
One CREATE TABLE … AS SELECT copies the result into your Iceberg lake, so the next test reads it fast.
Schedule the refresh
Turn the same SQL into a scheduled job that loads new rows every morning. No DBT project needed.
Hand over the SQL
When data engineering builds the pipeline, they start from a query that has already proved its value.
Or ask your agent
Claude or ChatGPT can explore the new file, join it, and schedule the load for you over MCP.
