Unity Catalog was designed for the open lakehouse. Beforehand, knowledge groups had been caught in silos, typically pressured to duplicate knowledge throughout platforms simply to make use of the instruments they needed. Each new platform or instrument meant copying datasets, rebuilding entry insurance policies from scratch, and protecting every part in sync. The outcome was elevated prices from redundant storage, insurance policies that drifted out of sync, and fragmented knowledge entry and discovery.
Once we open sourced Unity Catalog and launched Open APIs, we broke down the silos that beforehand saved clients locked-in. Enterprises may lastly maintain one copy of knowledge, use any compute engine, and govern every part from one place. The UC ecosystem has thrived since. At present, 1000’s of consumers use Unity Catalog to control and entry Delta Lake and Apache Iceberg tables, with dozens of integrations within the rising Unity Catalog ecosystem — from Apache Spark and Trino to DuckDB and Confluent Tableflow.
Exterior Entry to Managed Tables, Now in Beta
UC managed tables are the place openness meets efficiency. These superior tables use Predictive Optimization and Liquid Clustering to mechanically tune knowledge layouts, run compaction and vacuuming, and maintain statistics recent — delivering as much as 20× quicker queries and 50% decrease storage prices, whereas staying absolutely accessible via open APIs.
Now in Beta, exterior engines, resembling Apache Spark, Flink, and DuckDB, can create and write to UC managed Delta tables with centralized governance and automated optimizations.
With the Beta, exterior engines can:
- Create managed tables — Get up new UC managed tables straight from an exterior engine.
- Batch learn and write — Learn and write to managed tables with full transactional security.
- Stream to and from managed tables — Use managed tables as each a streaming supply and sink, enabling end-to-end real-time pipelines on Apache Spark.
As a result of each operation flows via UC managed tables constructed on catalog commits, you get serialized commits that stop log corruption and full auditability of each learn and write. Predictive Optimization continues to run seamlessly, even on tables accessed by exterior engines. Catalog commits additionally lay the groundwork for options like multi-statement, multi-table transactions that require a centralized commit coordinator.
The thriving UC ecosystem is continuous to develop as engines develop help for exterior entry to managed tables. Delta Kernel — the open supply Java and Rust library for studying, writing, and committing to Delta tables — abstracts the low-level protocol particulars so connector builders can concentrate on UC integration, not Delta implementation. Apache Spark, Delta Flink, and DuckDB have all leveraged Delta Kernel to help exterior writes to UC managed tables and combine with catalog-managed commits, and the ecosystem continues to develop. By dealing with the low-level protocol complexity, Delta Kernel makes it simple for any engine to combine with Unity Catalog which contributes to a rising ecosystem of connectors.
Safe Exterior Entry Made Attainable By Credential Merchandising
For an exterior engine to entry knowledge in UC, it wants a safe solution to authenticate and get scoped entry to cloud storage with out requiring broad, static permissions or credentials tied to a particular account. Unity Catalog handles this via credential merchandising, which is now typically out there (GA): UC points short-lived, scoped credentials to exterior engines on demand, with entry insurance policies enforced centrally.
Hundreds of consumers have used UC Open APIs and two additions make it production-ready at enterprise scale. Exterior engines can now authenticate to UC utilizing machine-to-machine (M2M) OAuth, assembly enterprise safety necessities with out counting on personalised entry tokens (PATs), that are per-user, long-lived, and arduous to rotate. And credentials are refreshed mechanically by engines through the UC credential merchandising APIs, so pipelines that run for hours full reliably with out tokens expiring mid-job.
With credential merchandising, enterprises can learn, write, and create managed and exterior tables in Unity Catalog from any appropriate engine or instrument. These credentials are short-lived, scoped to the requested useful resource, and ruled by UC privileges. This implies your platform crew retains full management over which principals can entry knowledge externally and what they’ll do with it.
With Unity Catalog’s Open APIs, we have empowered our groups to make use of their most popular instruments whereas sustaining governance and knowledge consistency. We will leverage the advantages of managed tables inside a really interoperable knowledge and AI platform that works throughout a number of compute engines.— Sudipta Das, Director of Enterprise Knowledge Operations at PepsiCo
Credential Merchandising for Volumes
Credential merchandising extends not solely to tables but additionally unstructured knowledge. Quantity credential merchandising is now in Public Preview, so exterior shoppers can request non permanent, scoped credentials to entry photographs, PDFs, and movies saved in volumes with Unity Catalog governance. The identical entry management mannequin, audit path, and scoped credentials apply whether or not you are querying a desk or processing a uncooked video file externally.
What’s Subsequent?
We’re persevering with to spend money on making exterior entry extra succesful. Credential merchandising at the moment governs coarse-grained entry controls for exterior engines. We have additionally developed performance to implement attribute-based entry controls (ABAC) for exterior reads, which makes governance fine-grained. This makes it potential to implement row and column stage ABAC insurance policies when UC managed tables are learn rom exterior engines.
Get Began At present
To get began with credential merchandising, see our documentation. To make use of the Beta of exterior entry to managed Delta tables:
- Enroll in “Exterior Entry to Unity Catalog Managed Delta Desk” within the Databricks preview portal (see Handle Databricks previews)
- Allow exterior knowledge entry in your metastore and grant EXTERNAL_USE_SCHEMA on the schema containing the tables you wish to entry.
- Create a brand new UC managed desk. To maneuver present knowledge, see the migration information for changing exterior tables to managed.
- Use Delta-Spark 4.2 with Unity Catalog 0.4.1 to create, learn, and write to managed tables from exterior compute. See the exterior entry documentation.
Be part of us at Knowledge and AI Summit 2026
Knowledge and AI Summit 2026 is sort of right here! Be part of us June 15-18, 2026 on the Moscone Heart in San Francisco, California to find out how main organizations are utilizing Unity Catalog to control knowledge and AI throughout engines. Register at the moment to get a primary have a look at what’s coming subsequent for open, unified governance.