The Period of Open Governance
A yr after we open-sourced Unity Catalog (UC), the outcomes are clear: openness isn’t only a precept, it’s working in apply.
Since then, a whole bunch of enterprises have adopted Unity Catalog as their basis for open, interoperable governance throughout Delta Lake, Apache Iceberg, and each main engine within the trendy information stack. What started as a dedication to open requirements has advanced right into a thriving ecosystem of open APIs, associate integrations, and buyer influence at scale.
Right this moment, Unity Catalog stands as essentially the most extensively adopted open catalog for information and AI. Information groups now not need to make trade-offs throughout efficiency, interoperability, and governance; they will have all of it.
From “why open?” to “open at scale”
In 2024, we open-sourced UC and launched UC Open APIs to reinforce interoperability with exterior instruments. These APIs make it easy for any engine to securely hook up with Unity Catalog, learn or write Delta and Iceberg tables, and apply governance mechanically by way of credential merchandising and centralized entry insurance policies.
A yr later, the ecosystem speaks for itself:
- Over 700 corporations now use UC to centralize governance throughout a number of engines and instruments.
- UC shopper SDKs see greater than 1 million downloads monthly, fueling cross-platform adoption.
- Companions like Starburst, ClickHouse, and Confluent have constructed deep integrations on high of UC Open APIs, extending governance properly past Databricks.
This momentum proves that interoperability scales greatest when openness and governance work collectively.
The very best catalog for Delta Lake and Apache Iceberg
Unity Catalog offers first-class assist for Delta and Iceberg throughout governance, entry, and efficiency. By means of UC Open APIs and the Iceberg REST Catalog API, organizations can securely join any engine by way of studying, writing, and creating tables whereas adhering to unified entry insurance policies.
Unity Catalog makes exterior entry easy:
- Outline entry controls as soon as, and UC mechanically enforces them throughout clouds, engines, and codecs.
- Credential merchandising points non permanent, scoped credentials behind the scenes, eradicating the necessity to configure cloud storage permissions or replicate insurance policies manually.
- Prolong governance to AI by way of a unified management airplane that lets you handle datasets, options, and mannequin variations.
- Join any software or engine, from Spark and Trino to customized ML pipelines, by way of Unity REST APIs and the Iceberg REST Catalog API.
Take governance one step additional by leveraging UC Managed Tables, the place openness meets efficiency. These Databricks-optimized tables use Predictive Optimization and Liquid Clustering to ship as much as 20× quicker queries and 50% decrease storage prices whereas staying totally open and accessible by way of commonplace APIs. Managed Tables symbolize the brand new commonplace: centralized governance, open codecs, and clever efficiency—multi functional.
Trade and ecosystem momentum behind UC Open APIs
Over the previous yr, UC Open APIs have helped a whole bunch of organizations break format silos, unify governance, and lengthen interoperability throughout each a part of their stack.
PepsiCo: Unified information governance throughout multi-engine analytics
PepsiCo runs a various analytics ecosystem with a number of compute engines, corresponding to Spark on Kubernetes. Traditionally, these engines needed to bypass UC and hook up with exterior tables by way of path-based entry. With UC Open APIs, PepsiCo can now undertake managed tables and have exterior engines entry information by way of a single, centralized governance with out requiring storage-level workarounds.
With Unity Catalog’s Open APIs, we have empowered our groups to make use of their most popular instruments whereas sustaining governance and information consistency. We are able to leverage the advantages of managed tables inside a very interoperable information and AI platform that works throughout a number of compute engines.— Sudipta Das, Director Enterprise Information Operations
Coinbase: Graph queries at scale with credential merchandising
Coinbase depends on PuppyGraph to course of terabytes of knowledge each day. UC Open APIs and credential merchandising get rid of the necessity for ETL pipelines, letting Coinbase question Delta and Iceberg tables instantly whereas imposing insurance policies and capturing audit logs.
Utilizing Unity Catalog’s Open APIs, PuppyGraph can question 2TB+ of knowledge each day with non permanent credentials, analyzing service dependencies at scale – all whereas protecting governance centralized in UC.— Eric Solar, Head of Information Platform at Coinbase
Ecosystem partnerships
Unity Catalog sits on the coronary heart of a rising ecosystem of companions, extending governance past Databricks:
- Confluent Tableflow integrates with UC to transform Kafka occasion streams into Delta tables. These tables are accessible by way of UC Open APIs and have governance insurance policies mechanically utilized.
- ClickHouse permits its customers to leverage UC Open APIs to find and question Delta and Iceberg tables for real-time analytics and observability. With this integration, customers can entry Delta and Iceberg tables ruled by UC instantly from ClickHouse, whereas protecting UC on the heart of governance.
- Starburst Trino has developed a Delta Lake connector that helps studying managed tables ruled by Unity Catalog. To assist writes to managed tables, Starburst has additionally built-in with the non-public preview of exterior writes by way of UC Open APIs.
Starburst shares Databricks’ imaginative and prescient of openness and interoperability throughout the information ecosystem. By integrating with Databricks Unity Catalog, we’re enabling prospects to create a single supply of fact for all their information, with centralized governance and the pliability to leverage the instruments of their selection.— Justin Borgman, CEO, Starburst Information
What’s Subsequent for UC Open Connectivity?
Unity Catalog continues to evolve as essentially the most open and interoperable governance layer for the lakehouse. Right here’s what’s coming subsequent:
- Exterior writes and desk creation for UC-managed tables: Right this moment, exterior engines can learn UC-managed tables. Coming quickly to Public Preview, we’ll allow exterior writes by way of Unity REST APIs and desk creation instantly from exterior shoppers. Focused on testing this out? Be part of our Non-public Preview.
- Word: We already supply full assist for the Iceberg REST Catalog API, permitting exterior engines to learn (Usually Out there) and write (Public Preview) to Unity Catalog–managed Iceberg tables
- Safe entry past tables: Credential merchandising is being prolonged to volumes, so unstructured information could be securely accessed from instruments like Daft and Ray for AI/ML workflows.
- Simply migrate to UC Managed Tables: With just a few easy instructions, you’ll be capable to convert UC exterior or catalog-federated overseas tables to completely managed tables—preserving Delta historical past, settings, permissions, and views.
- Unity Catalog 0.4 launch: within the subsequent launch of UC (v0.4), we’re including managed tables assist and implementing credential renewal within the UC Spark shopper for lengthy operating jobs.
Get Began Right this moment
Unity Catalog Open APIs can be found for each Delta and Iceberg shoppers. You can begin by:
Begin constructing with UC Open APIs right this moment and see how simple interoperability and unified governance could be. To get began with Unity Catalog, comply with the guides for AWS, Azure, and GCP.