Twenty years of building data platforms has taught me one thing above all: nobody fights over the storage layer until it’s too late. Everyone argues about the query engine, the transformation framework, the warehouse vs. lakehouse decision. Then six months in, some team realizes they’re completely locked into one vendor because the catalog doesn’t talk to anything else, and the migration cost becomes the kind of number that ends careers.
That lesson is playing out right now across the industry, and the stakes have never been higher. Apache Iceberg has effectively won the open table format war. Iceberg v3 went GA in May 2026 (production-ready with the 1.11.0 release), and every major engine from Spark to Flink to Trino supports it natively. The format question is settled. The question nobody is answering loudly enough is: what catalog are you using to manage those Iceberg tables, and will it let you escape if you need to?
This article is about that catalog layer: what the Iceberg REST catalog specification actually is, why it matters more than the table format itself, and how to choose between the three serious open-source implementations competing for this space right now.
Why the Catalog Layer Is the Real Lock-In
When most people learn about Iceberg, they focus on what makes the table format clever: hidden partitioning, time travel, schema evolution without rewrites, the snapshot-based commit model that enables atomic operations on object storage. That’s all real and genuinely useful. I wrote about it in our Apache Iceberg and the data lakehouse guide.
What they miss is that none of that works without a catalog. The catalog is the service that tracks where tables live, what schemas they have, which snapshot is current, and who has permission to read or write them. Without a catalog, Iceberg tables are just a pile of Parquet files with some metadata JSON sitting in S3. The catalog is the nervous system.
For years, most teams used either the Hive Metastore (HMS) or a vendor-proprietary catalog (Snowflake’s internal catalog, Databricks’ Hive Metastore compatibility layer, Glue for AWS-native stacks). The problem with HMS is that it requires a Thrift connection, which means your compute and catalog need to be on the same network and often the same cloud. The problem with proprietary catalogs is obvious.
The Iceberg REST Catalog specification changed this equation. It defines a standardized HTTP/REST API for catalog operations: create namespace, create table, load table, update table, and commit transaction. Any engine that speaks REST can talk to any compliant catalog. Spark running in your Kubernetes cluster can write to a catalog hosted on the opposite coast. A Trino cluster managed by your platform team can read tables created by a Flink pipeline maintained by a different team entirely. This is the promise.
The spec covers three major areas: table management (CRUD operations on tables and namespaces), credential vending (the catalog generates short-lived cloud credentials scoped to specific table paths), and multi-table transactions (commit coordination across tables in a single atomic operation). I want to spend real time on credential vending because it’s the feature most teams underestimate.
Credential Vending: The Security Architecture Nobody Explains
Here is the problem credential vending solves. You have a data lake on S3. Your compute engines need to read and write files. The naive approach is to give every engine broad IAM permissions: s3:GetObject and s3:PutObject on the entire bucket. This is a disaster. A compromised Spark cluster or a misconfigured job can read every table in the lake, including tables from other teams with different sensitivity levels.
The REST catalog specification’s credential vending model inverts this. When a compute engine requests a table from the catalog, the catalog generates short-lived AWS STS session tokens (or equivalent GCS/Azure credentials) scoped to exactly the S3 prefixes that table uses. The engine gets credentials good for maybe fifteen minutes that can only touch the specific path for the table it asked for. When the job finishes or the credentials expire, access evaporates automatically.
This is how enterprises with serious security requirements should be running their data lakes, and it’s been impossible to do cleanly until the REST catalog spec formalized it. Apache Polaris 1.4.0 (the first release after its graduation to Apache Software Foundation Top-Level Project in February 2026) added AWS STS session tags for CloudTrail correlation, so you get not just access scoping but auditable attribution in CloudTrail logs linking every S3 operation back to the catalog request that produced the credentials.
For teams building multi-engine lakehouse architectures, this changes the security posture fundamentally. Instead of managing a complex IAM policy matrix for every engine and every table path, you manage access control at the catalog layer. The catalog becomes your authorization plane, and the cloud IAM policies become a simple blanket grant to the catalog’s service role. This pairs naturally with the kind of data governance work I cover in our data contracts and data quality ownership guide.
The Three Serious Open-Source Catalogs
Apache Polaris: The Standard-Bearer
Apache Polaris graduated to an Apache Software Foundation Top-Level Project on February 18, 2026. This matters because TLP graduation means the project passed ASF’s community health and governance reviews: there is a functioning PMC, broad committer diversity, documented release processes. It is not a project controlled by a single vendor.
Polaris was originally created and donated by Snowflake. That history is worth knowing because it explains some of Polaris’s design decisions, particularly around its multi-catalog federation model and its treatment of external catalogs. Snowflake needed a way to expose its internal Iceberg tables to outside engines, and Polaris was the solution. What Snowflake donated became genuinely more useful as an open standard than as a Snowflake proprietary thing, so they open-sourced it.
The core Polaris architecture is straightforward. You run a Polaris server (it ships as a Docker image or you deploy it on Kubernetes), configure a persistence backend (PostgreSQL by default, CockroachDB added in the 1.4.0 release for teams that want distributed persistence), and point your query engines at it. The server implements the full Iceberg REST catalog API plus the credential vending spec. You manage access through principal roles and catalog grants, which form a hierarchy: principals (users or service accounts) get assigned to roles, roles get granted permissions on catalogs, namespaces, or tables.
Polaris has a concept called “service connections” for cloud storage, where you configure the trust relationship between Polaris and your cloud provider at setup time. After that, credential vending is automatic. The Polaris server assumes an IAM role, uses it to call STS, and vends scoped tokens to engines. The engine never needs a long-lived credential.
The main limitation of Polaris today is that multi-table transactions (atomic commits across multiple tables) are still in development. Polaris handles single-table commits atomically, which covers the vast majority of use cases, but if you need cross-table atomic operations as a first-class primitive, you are looking at Nessie.
Unity Catalog: Databricks’s Open Play
Databricks open-sourced Unity Catalog under the Apache 2.0 license and placed it under the LF AI & Data Foundation as a sandbox project. If you have spent any time in the Databricks ecosystem, you know Unity Catalog as the managed service that replaced the old workspace-scoped Hive Metastore. Open-source Unity Catalog is the same metadata model, accessible outside Databricks managed infrastructure.
What makes Unity Catalog interesting is its ambition. It is not just an Iceberg catalog. It is a unified catalog for Delta Lake tables, Iceberg tables, Hudi tables, volumes (unstructured data), functions, and ML models. The data model unifies governance across formats, which matters for organizations that are not yet fully on Iceberg or that run Delta Lake for Databricks workloads and Iceberg for everything else.
Catalog Commits, the cross-table transaction feature in open-source Unity Catalog, went GA in 2026. This allows coordinating commits across multiple tables in a single atomic operation, which is the capability that makes large-scale ETL pipelines tractable without complex compensating transaction logic.
The honest concern with Unity Catalog is the governance question. Despite the Apache 2.0 license and the LF AI & Data foundation involvement, this is fundamentally a project that Databricks controls and is deeply invested in commercially. The open-source version exists partly because it makes Databricks’ commercial offering more attractive as a hub in a multi-engine lakehouse. That is not a reason to avoid it, but it is a reason to watch the project’s governance closely over the next few years. A project that a single company controls can have its direction changed without community consensus.
For teams already invested in the Databricks ecosystem, open-source Unity Catalog as a catalog for non-Databricks engines is a compelling path. You get a consistent governance model across your Databricks and non-Databricks workloads, and Catalog Commits handles the cross-table transaction requirements that Polaris does not yet fully address. The cloud data warehouse comparison I wrote earlier covers the broader Databricks platform decision.
Project Nessie: Git Semantics for Data
Project Nessie takes a different angle entirely. Where Polaris and Unity Catalog are primarily access-management and interoperability plays, Nessie is about bringing Git-style branching and commit semantics to your data lake catalog. Nessie was created by Dremio and is now an open-source project independent of Dremio’s commercial offerings.
The mental model is: your catalog is a Git repository. Tables are files. Namespaces are directories. You can create branches, make commits on branches, merge branches, and tag specific versions. The commit history is the audit log. A branch is an isolated environment where you can run experimental ETL jobs without affecting the main branch, then merge the changes atomically when they look right.
For data engineering teams, this is genuinely transformative for development workflows. Consider the pattern: you create a branch for a migration job, run the job on the branch, validate that the output tables look correct, then merge to main. The merge is atomic, so downstream consumers never see partially-migrated data. Compare this to the typical migration approach, which involves temporary tables, rename operations, and a tense period where you hope nothing reads the table between steps.
Nessie also solves the multi-table atomic commit problem that Polaris is still working on, because the branch model handles it naturally. Commit all your table changes on a branch, then merge. The merge operation is atomic across all tables in the commit.
The tradeoff with Nessie is operational complexity and ecosystem support. Nessie requires that your query engines support the Nessie-specific extensions to the Iceberg catalog API (specifically the reference parameter that specifies which branch or tag to read). Most major engines do support this now (Spark, Trino, Flink, Dremio, and others), but not every tool does. If you are using a BI tool that connects directly to Iceberg tables via the REST catalog, verify Nessie compatibility before committing.
Nessie’s claims about throughput (the project documentation cites millions of tables and thousands of commits per second capability) are vendor-published claims from Dremio, not independently verified benchmarks. I treat them as a signal that the system is designed for scale, not as a guaranteed specification.
AWS Glue: The Managed Option
No honest comparison of Iceberg catalogs is complete without acknowledging AWS Glue. For teams running entirely on AWS, Glue has added Iceberg REST catalog support, and it means you get a fully managed, serverless catalog with no infrastructure to operate. You pay for catalog API calls and storage, and AWS handles availability, backups, and scaling.
The limitation is obvious: Glue ties you to AWS. The REST catalog spec means your engines can run anywhere and talk to Glue, but the catalog itself lives in one cloud. For multi-cloud architectures or teams that want genuine portability, a self-hosted Polaris or Nessie running on Kubernetes gives you more control. For AWS-native shops that have already accepted the hyperscaler dependency, Glue is the path of least operational resistance.
What Iceberg v3 Changes
Apache Iceberg v3 became production-ready with the 1.11.0 release in May 2026. The most significant addition for practical purposes is the VARIANT data type, which stores semi-structured and JSON-like data natively in Iceberg tables without schema flattening. This has been a consistent pain point for teams ingesting event streams or webhook payloads with variable schemas. Previously, you had three bad options: flatten everything into hundreds of columns (schema hell), store as a string and parse at query time (performance hell), or use a separate document store (operational hell).
VARIANT addresses this by representing semi-structured data as a typed columnar format that query engines can push predicates into without parsing the entire document. Snowflake pushed this feature through the spec, and it shows in Polaris’s early support for it.
The catalog implication of v3 is that your catalog implementation needs to understand the new table metadata fields. Apache Polaris has v3 support in progress, with Snowflake’s commercial catalog ahead given that Snowflake GA’d Iceberg v3 on their platform in May 2026. Open-source Unity Catalog and Nessie have stated v3 compatibility as a priority. If you are evaluating catalogs today for a table format migration to v3, verify v3 support in the specific version of the catalog you are deploying.
Choosing Your Catalog: The Decision Framework
After years of building data platforms, I have a simple set of questions I work through when making catalog decisions.
What is your primary cloud? If you are AWS-native and do not need multi-cloud portability, Glue is worth evaluating seriously before you take on the operational overhead of a self-hosted catalog. If you are multi-cloud or care about avoiding cloud lock-in, Polaris or Nessie on Kubernetes.
Do you need cross-table atomic commits today? If yes, Nessie or Unity Catalog. If single-table atomicity is sufficient (which covers most OLAP workloads), Polaris is mature and has a strong ecosystem story.
Are you already in the Databricks ecosystem? Unity Catalog as a catalog for non-Databricks engines gives you governance consistency. Just watch the governance question as the project matures.
Do your teams use isolation environments for data development? If development workflows would benefit from branch-based data isolation (data engineers building in a feature branch, data scientists experimenting without breaking production datasets), Nessie’s model is genuinely compelling and unlike anything the other catalogs offer. See our data pipeline orchestration guide for how this fits into broader pipeline workflows.
What query engines are you running? Most modern engines (Spark, Flink, Trino, Dremio, StarRocks, DuckDB) support the Iceberg REST spec and will work with any of the three open-source catalogs. Verify your specific engine versions and the Nessie-specific extensions if you go that route.
Deployment Patterns in Practice
For teams deploying Polaris on Kubernetes, the setup is relatively straightforward. The Polaris Docker image is the starting point. You configure a PostgreSQL backend (or CockroachDB for distributed durability), configure the cloud storage integration, and set up the trust relationship between Polaris and your cloud provider’s IAM. The Polaris documentation recommends running behind a service mesh for mTLS between engines and the catalog server, which is sensible for production deployments.
The Polaris data model has a concept of “catalog” as a top-level entity (not to be confused with the catalog server itself). You might have a prod-catalog and a dev-catalog on the same Polaris server, each with separate storage configurations and access control hierarchies. This maps cleanly to the multi-environment patterns most platform teams already use.
For Nessie on Kubernetes, the operator model is available. Nessie uses RocksDB for embedded storage at small scale, or you can configure it with PostgreSQL or CockroachDB for production workloads. The configuration surfaces are similar to Polaris, but Nessie adds the reference management API that your pipelines need to use to specify branches and tags.
One pattern I have seen work well for teams migrating from Hive Metastore is running Polaris or Nessie alongside the existing HMS initially, with new tables going to the REST catalog and a gradual migration path for legacy tables. The REST catalog spec handles this cleanly because engines can be configured with multiple catalogs simultaneously. The data catalog and lineage tools like OpenMetadata and DataHub have integrations for both REST catalogs and HMS, so your metadata discovery layer does not need to change during migration.
The Interoperability Dividend
Here is the actual value proposition that gets missed in all the vendor positioning: when your catalog is a standard REST API, your data governance and access control become a catalog concern rather than an engine concern. You do not configure access control separately in Spark, Flink, Trino, and your BI tool. You configure it once in the catalog, and every engine that goes through the catalog gets the same access control applied.
This is the enterprise governance model that people have been trying to build with HMS and manual IAM policies for fifteen years. It mostly did not work because the catalog and the access control were not integrated. The REST catalog spec, combined with credential vending, finally makes it work cleanly.
For teams that have built data mesh architectures (see our data mesh architecture guide), the REST catalog with proper credential vending becomes the technical mechanism that enables domain ownership without surrendering governance. Each domain team runs with credentials scoped to their tables. They can share read access to their data products by granting catalog roles to other teams’ principals. The catalog enforces the contracts. dbt integrations with REST catalog are maturing quickly, which means transformation pipelines get the same access control as query engines.
What I Would Deploy Today
If I were starting a new lakehouse build today, I would default to Apache Polaris for the following reasons: it is a genuine Apache Software Foundation project with real community governance, it has the strongest credential vending implementation, it has good support from most major engines, and its production story is cleaner than Nessie’s additional complexity. Polaris 1.4.0’s support for CockroachDB as a persistence backend is useful for teams that need distributed catalog durability without managing a PostgreSQL cluster.
I would choose Nessie if development workflow isolation is a genuine pain point for my data engineering team. The branch-based model is genuinely better for environments where you have many engineers or teams making catalog changes concurrently. The operational complexity is real but manageable on Kubernetes, and the Dremio team has been actively maintaining it for years.
I would choose open-source Unity Catalog only if I was already deeply in the Databricks ecosystem and needed governance consistency across Databricks and non-Databricks workloads. The project is capable, but the single-vendor governance reality is a risk I would monitor.
I would choose Glue if I was AWS-native, operationally conservative, and willing to accept cloud lock-in for zero catalog infrastructure overhead.
Whatever you choose, verify your engines’ support for the Iceberg REST catalog spec version you need, verify credential vending support if you care about access scoping (you should), and do not let the catalog decision be an afterthought. The catalog is the governance layer. Everything else is plumbing.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.