DuckLake: The Open Table Format That Stores Metadata in SQL and What That Changes for Data Teams

I have spent twenty years building data infrastructure, and one pattern plays out with almost eerie consistency: the tooling you adopt to solve one problem creates a different problem you did not see coming. Hadoop solved the “I have too much data for one machine” problem and gave us the joy of managing HDFS namenode memory. Iceberg solved the “Hive partitions are a footgun” problem and gave us a metadata file explosion that requires its own compaction strategy. Every generation pays a tax for the convenience of the previous one.

That context is why I find DuckLake interesting. The DuckDB Labs team released version 1.0 in April 2026, and the core idea is genuinely different from every table format that came before it: stop storing table metadata in object storage files and put it in a SQL database instead. It sounds almost too obvious once you hear it, and yet nobody building a data lakehouse format had done it before.

I want to walk through what DuckLake actually is, where it fits against Apache Iceberg and the existing open table formats, and where the real tradeoffs live for teams deciding whether to adopt it.

The Metadata File Problem Nobody Likes Talking About

If you have operated an Apache Iceberg table at anything beyond toy scale, you know the metadata sprawl problem firsthand. Every time you write to an Iceberg table, the format creates new metadata files: a snapshot JSON file, a manifest list, manifest files pointing at data files. Over time, especially with frequent small writes, you accumulate thousands of tiny files in your metadata layer.

This is not a hypothetical concern. I have seen production Iceberg setups where the metadata directory held more objects than the data directory. That makes listing operations slow, increases object storage API costs, and causes catalog operations to become the bottleneck on write paths that were supposed to be fast. Iceberg has compaction guidance and hidden metadata cleanup procedures for a reason.

The root of the problem is that object storage was designed for large files, not for high-frequency small-file writes. Writing metadata as files to S3 or GCS means you are using a system with no real transactions, limited consistency guarantees, and high per-operation overhead to do something that relational databases have been doing efficiently for decades.

DuckLake’s answer is to not do that at all.

What DuckLake Actually Is

DuckLake separates cleanly into three components: catalog, storage, and compute.

The catalog is a SQL database. All table metadata: schema definitions, partition information, snapshots, time-travel history, column statistics, everything that Iceberg would write as JSON and Avro files on object storage, lives instead in relational tables inside a SQL database. As of v1.0, the supported catalog databases are SQLite, PostgreSQL, and DuckDB itself. The DuckLake specification defines exactly which tables the catalog needs and what types they use, so any conforming SQL database can serve as a catalog without a separate catalog service like Polaris or Unity Catalog.

The storage layer is Parquet files on object storage, exactly like Iceberg. If you already have an Iceberg or Delta Lake setup, the underlying file format is the same. DuckLake uses deletion vectors (compatible with Iceberg’s deletion vector format) for row-level deletes without immediately rewriting files.

The compute layer is pluggable. DuckDB is the primary engine and gets the most optimized integration, but Trino, Apache Spark, and Apache DataFusion can also read DuckLake tables. Multi-engine support was a design requirement from the start, because the DuckDB Labs team understood that no single engine wins every workload.

The key insight is that the catalog is now a proper database with ACID transactions, primary keys, indexing, and concurrency control. When you commit a write to a DuckLake table, you are doing a SQL transaction against the catalog database, not atomically swapping pointer files in object storage. That changes what becomes easy and what becomes hard.

v1.0: What Shipped in Production

DuckLake went from concept to a first stable specification in roughly a year. The v1.0 release in April 2026 carries a stability guarantee: backward compatibility is maintained going forward, which is the signal that a format is ready for production data.

The feature list in 1.0 addresses real operational concerns:

Data inlining handles small inserts, updates, and deletes by embedding them directly in the catalog database rather than creating new Parquet files on object storage. If you are writing one row at a time to a DuckLake table (something that makes Iceberg sob quietly), inlining prevents the small-file accumulation problem from starting. The catalog database can hold the inline data until a threshold is reached, then flush it to a Parquet file in a background compaction step. You stop paying per-operation file creation costs for small writes.

Sorted tables allow specifying sort keys on a table, and the format maintains the sort order during compaction. This enables DuckDB to use the sort order for filter pushdown and range pruning without reading full files. For analytical workloads that query by date ranges or identifier ranges, this is a meaningful query acceleration.

Bucket partitioning addresses high-cardinality columns. Iceberg partitions on exact column values, which works well for date columns but creates partition explosion for columns like user_id. Bucket partitioning hashes the column into a fixed number of buckets, giving you data co-location for joins and aggregations without the cardinality problem.

Deletion vectors compatible with Iceberg’s deletion vector format are included. This means row-level deletes without file rewrites, and it means the deletion vector format is one point of interoperability between DuckLake and Iceberg if you need to bridge between them.

Checkpoints can now run concurrently with reads, insertions, and deletions. Earlier versions had contention where checkpoint operations could block query execution. The v1.0 concurrency model fixes that.

The Catalog Is What Changes Everything

Understanding DuckLake requires sitting with what it means to have metadata in a SQL database rather than in files.

Listing partitions in Iceberg means reading object storage: list the metadata directory, parse the manifest list, read manifests, extract partition info. This is multiple round trips against an eventually consistent storage system, and the cost scales with the number of snapshots and manifests accumulated. For a table with years of write history and no aggressive cleanup, this can be the slow part of query planning.

Listing partitions in DuckLake is a SQL query against a table with a primary key and an index. It is fast regardless of how many historical snapshots exist, because the catalog database is optimized for exactly this kind of lookup.

Multi-table ACID transactions become possible in a way they are not with Iceberg. You can atomically update two DuckLake tables in the same catalog database with a single SQL transaction. Iceberg has a concept of multi-table transactions through the Iceberg REST catalog specification, but the catalog has to implement it, and consistency relies on optimistic concurrency with retries. DuckLake gets it for free from the underlying database’s transaction semantics.

Catalog management for data lakehouses is a nontrivial infrastructure concern with Iceberg: you need to run Polaris, Unity Catalog, Nessie, or AWS Glue, and each has its own operational surface area. DuckLake’s catalog requirement is just “a SQL database,” something every team already operates.

DuckDB Labs has published benchmark results claiming 10x faster queries and 10x more transactions per second compared to Iceberg in their internal tests, and a separate streaming benchmark showing 926x faster reads and 105x faster ingestion versus Iceberg on a streaming workload. I want to be clear: these are vendor-published benchmarks from the team building DuckLake, not third-party reproductions. I always treat vendor benchmark numbers as evidence of potential, not as settled facts. That said, the architectural reasoning for why metadata operations would be faster is sound, and the DuckLake community is actively sharing independent results.

Where DuckLake Wins and Where It Does Not

The case for DuckLake is strongest in specific scenarios.

High-frequency small writes. If you are streaming data into a table with frequent small batches, Iceberg’s file-based model creates operational headaches. Data inlining and a SQL-based commit path make DuckLake significantly more write-friendly for this pattern.

Small-to-medium data engineering teams. Running Iceberg at production scale means operating Polaris or another catalog service, understanding Iceberg’s hidden metadata, scheduling compaction jobs, and keeping catalog-compute compatibility matrices in your head. DuckLake’s requirements are simpler: a PostgreSQL instance (which you almost certainly already have) and Parquet on object storage. That is a meaningful operational reduction.

Analytical workloads that live primarily in DuckDB. If DuckDB is already your analytical engine and you are using it for interactive analysis, data transformation, and ad hoc queries, DuckLake gives you a lakehouse format that is deeply optimized for DuckDB’s execution model. Sorted tables, filter pushdown through the SQL catalog, and inlined small rows play directly into how DuckDB runs queries.

Teams migrating off Iceberg’s operational complexity. The deletion vector format compatibility means you can convert Iceberg tables to DuckLake without a full data rewrite in some scenarios. The Parquet file format is identical.

The case against DuckLake is also real.

The catalog database becomes a single point of failure. In Iceberg, the catalog service can be stateless (the state lives in object storage), which makes it easier to run at high availability. With DuckLake, the SQL catalog database holds state that matters. If you use SQLite as your catalog, you have no distributed failover. If you use PostgreSQL, you need to run a production-grade PostgreSQL instance with replication and backup. For most teams, this is not a new problem, but it is a dependency that does not exist with a file-based format.

Spark and other engine support is still maturing. DuckDB integration is first-class. Trino and Spark support exists, but it is less mature than their Iceberg integration. For shops where Apache Spark is the primary compute engine running petabyte-scale jobs, Iceberg’s deep Spark integration is a real advantage that DuckLake has not fully closed.

The ecosystem is young. Iceberg has catalog services, Terraform modules, cloud-managed offerings from every major vendor, and years of community tooling. DuckLake’s ecosystem is small. Data catalog and lineage tools like OpenMetadata and DataHub do not yet have native DuckLake integration.

Petabyte scale is unproven in the field. MotherDuck announced managed DuckLake support as production-ready for petabyte scale in April 2026, but “announced production-ready” and “proven by independent production deployments at petabyte scale” are different things. Teams running truly enormous datasets on Iceberg with years of stability evidence are right to be cautious.

Getting Started in Practice

For a team that wants to try DuckLake without committing to a migration, the local setup is genuinely simple. Install DuckDB and the DuckLake extension:

INSTALL ducklake FROM community;
LOAD ducklake;

Create a catalog backed by a local PostgreSQL database and attach it:

ATTACH 'ducklake:postgresql://user:pass@localhost/my_catalog'
    AS lake (TYPE DUCKLAKE, DATA_PATH 's3://my-bucket/lake/');

From that point, you use standard SQL to create and query tables in the DuckLake catalog:

CREATE TABLE lake.events (
    event_id BIGINT,
    user_id BIGINT,
    event_type VARCHAR,
    ts TIMESTAMP
) SORTED BY (ts);

INSERT INTO lake.events SELECT * FROM read_parquet('s3://source-bucket/events/*.parquet');

SELECT event_type, count(*) as n
FROM lake.events
WHERE ts >= TIMESTAMP '2026-01-01'
GROUP BY 1
ORDER BY 2 DESC;

The table metadata lives in the my_catalog PostgreSQL database. The Parquet files go to s3://my-bucket/lake/. DuckDB manages the coordination.

For time travel, DuckLake surfaces the snapshot history through the catalog:

SELECT * FROM lake.events AT (VERSION => 42);
SELECT * FROM lake.events AT (TIMESTAMP => TIMESTAMP '2026-08-01 00:00:00');

This looks identical to Iceberg’s time travel syntax in most query engines, which is intentional. The DuckLake specification was designed with SQL-standard time travel syntax as a first-class requirement.

MotherDuck and the Managed Path

MotherDuck, the managed DuckDB platform, added DuckLake 1.0 support alongside the format’s stable release. For teams that want the DuckLake model without operating the catalog database themselves, MotherDuck manages the catalog. Data files still live in your own object storage bucket, which means you retain ownership of the raw Parquet files and can read them independently.

This is a reasonable division: the operationally complex piece (the catalog database with its replication, backup, and access control) is managed, and the bulk data stays in storage you control. It is similar in spirit to how serverless databases handle the infrastructure complexity without eliminating data ownership.

The BYOC (bring your own cloud) model where DuckLake catalog runs in MotherDuck’s infrastructure but data files stay in your bucket also addresses the data residency concerns that prevent some regulated workloads from using fully managed services.

The Deeper Question: Do We Need Another Format?

A fair pushback on DuckLake is: the data lakehouse world just spent years converging on Iceberg as a standard. Apache Iceberg, Delta Lake, and Apache Hudi each had their moment, and the industry was gradually settling on Iceberg as the winner. Why introduce another format now?

The answer I find most compelling is that DuckLake is not trying to win the enterprise petabyte-scale Spark job category. It is targeting a different workload profile: teams that primarily use DuckDB for analytics, that have high-frequency write patterns that Iceberg handles poorly, and that want a simpler operational model than running a full catalog service.

The comparison I reach for is DuckDB itself versus Spark. DuckDB did not replace Spark. It found a workload category, in-process analytics on a single node up to terabyte scale, where Spark was genuinely overkill. DuckLake follows the same pattern: it is not Iceberg’s replacement, it is the format for workloads where Iceberg’s complexity and operational overhead are overkill.

The multi-engine story matters for this positioning. A DuckLake table can be queried by Trino and Spark today, and the specification is public enough that other engines can implement readers. The format is not betting everything on DuckDB as the only compute. That architectural decision is what separates it from a vendor lock-in play.

For teams using data pipeline orchestration with Airflow or Dagster to coordinate data flows into and out of a lakehouse, DuckLake’s simpler commit model means less operational state to track. You are not managing Iceberg’s optimistic concurrency retries; you are relying on the catalog database’s locking.

My Actual Recommendation

If you are already running Iceberg at scale and it is working, I would not migrate. The ecosystem maturity and multi-engine support in Iceberg is real, and a stable production system is worth a lot. What I would do is watch DuckLake’s ecosystem growth over the next six to twelve months and revisit when independent benchmarks and more production deployments fill in the picture.

If you are a new project or a team starting fresh with a lakehouse, DuckLake deserves a serious look, particularly if your primary engine is DuckDB. The operational simplicity is real, the format is stable, and the architectural reasoning for faster metadata operations is sound. Run your own benchmarks against your actual workload before committing.

If you are a team with high-frequency small-write patterns, DuckLake’s data inlining feature alone may be worth the evaluation. That is the scenario where Iceberg’s file model causes the most pain, and it is where DuckLake’s SQL catalog pays off most clearly.

The data engineering world has a habit of treating format decisions as permanent, but they are not. You can run a DuckLake catalog on PostgreSQL for your high-frequency write tables and maintain Iceberg for the batch workloads where the ecosystem tooling is better. The Parquet file format is the same under both. The format layer is more swappable than people treat it.

Twenty years of building infrastructure has taught me to be suspicious of anything that claims to solve everything, and I apply that same skepticism here. DuckLake solves a real class of problems with a genuinely different approach. It does not solve all problems. Pick the tool for the workload, and do not let format standardization debates substitute for actually understanding your write patterns and operational constraints.

Updated September 2026: Article written following DuckLake v1.0 release in April 2026. Performance benchmarks cited are vendor-published by DuckDB Labs and MotherDuck and not independently validated at publication time.