Unity Catalog Python user-defined features (UC Python UDFs) are more and more utilized in fashionable information warehousing, working thousands and thousands of queries each day throughout hundreds of organizations. These features permit customers to harness the complete energy of Python from any Unity Catalog-enabled compute, together with clusters, SQL warehouses and DLT.
We’re excited to announce a number of enhancements to UC Python UDFs that are actually accessible in Public Preview on AWS, Azure, and GCP with Unity Catalog clusters working Databricks Runtime 16.3, SQL warehouses (2025.15), and Serverless notebooks and workflows:
- Assist for customized Python dependencies, put in from Unity Catalog Volumes or exterior sources.
- Batch enter mode, providing extra flexibility and improved efficiency.
- Safe entry to exterior cloud providers utilizing Unity Catalog Service Credentials.
Every of those options unlocks new prospects for working with information and exterior methods instantly from SQL. Beneath, we’ll stroll by way of the main points and examples.
Utilizing customized dependencies in UC Python UDFs
Customers can now set up and use customized Python dependencies in UC Python UDFs. You possibly can set up these packages from PyPI, Unity Catalog Volumes, and blob storage. The instance operate beneath installs the pycryptodome from PyPI to return SHA3-256 hashes:
With this function, you’ll be able to outline steady Python environments, keep away from boilerplate code, and convey the capabilities of UC Python UDFs nearer to session-based PySpark UDFs. Dependency installations can be found beginning with Databricks Runtime 16.3, on SQL warehouses, and in Serverless notebooks and workflows.
Introducing Batch UC Python UDFs
UC Python UDFs now permit features to function on batches of knowledge, much like vectorized Python UDFs in PySpark. The brand new operate interface affords enhanced flexibility and offers a number of advantages:
- The batched execution offers customers extra flexibility: UDFs can maintain state between batches, i.e., carry out costly initialization work as soon as on startup.
- UDFs leveraging vectorized operations on pandas collection can enhance efficiency in comparison with row-at-a-time execution.
- As proven within the cloud operate name instance beneath, sending batched information to cloud providers may be cheaper than invoking them one row at a time.
Batch UC Python UDFs, now accessible on AWS, Azure, and GCP, are also referred to as Pandas UDFs or Vectorized Python UDFs. They’re launched by marking a UC Python UDF with PARAMETER STYLE PANDAS and specifying a HANDLER operate to be known as by title. The handler operate is a Python operate that receives an iterator of pandas Sequence, the place every pandas Sequence corresponds to 1 batch. The handler features are suitable with the pandas_udf API.
For example, think about the beneath UDF that calculates the inhabitants by state, based mostly on a JSON object mapping that it downloaded on startup:
Unity Catalog Service Credential entry
Customers can now leverage Unity Catalog service credentials in Batch UC Python UDFs to effectively and securely entry exterior cloud providers. This performance permits customers to work together with cloud providers instantly from SQL.
UC Service Credentials are ruled objects in Unity Catalog. They will present entry to any cloud service, resembling key-value shops, key administration providers, or cloud features. UC Service credentials can be found in all main clouds and are at present accessible from Batch UC Python UDFs. Assist for regular UC Python UDFs will observe sooner or later.
Service credentials can be found to Batch UC Python UDFs utilizing the CREDENTIALS clause within the UDF definition (AWS, Azure, GCP).
Instance: Calling a cloud operate from Batch UC Python UDFs
In our instance, we are going to name a cloud operate from a Batch UC Python UDF. This performance permits for seamless integration with current features and allows using any base container, programming language, or setting.
With Unity Catalog, we are able to implement efficient governance of each Service Credential and UDF objects. Within the determine above, Alice is the proprietor and definer of the UDF. Alice can grant EXECUTE permission for the UDF to Bob. When Bob calls the UDF, Unity Catalog Lakeguard will run the UDF with Alice’s service credential permissions whereas making certain that Bob can’t entry the service credential instantly. UDFs will use the defining consumer’s permissions to entry the credentials.
Whereas all three main clouds are supported, we are going to deal with AWS on this instance. Within the following, we are going to stroll by way of the steps to create and name the Lambda operate.
Making a UC service credential
As a prerequisite, we should arrange a UC Service Credential with the suitable permissions to execute Lambda features. For this, we observe the directions to arrange a service credential known as mycredential. Moreover, we permit our position to invoke features by attaching the AWSLambdaRole coverage.
Making a Lambda operate
Within the second step, we create an AWS Lambda operate by way of the AWS UI. Our instance Lambda HashValuesFunctionNode runs in nodejs20.x and computes a hash of its enter information:
Invoking a Lambda from a Batch UC Python UDFs
Within the third step, we are able to now write a Batch UC Python UDF that calls the Lambda operate. The UDF beneath makes the service credentials accessible by specifying them within the CREDENTIALS clause. The UDF invokes the Lambda operate for every enter batch, calling cloud features with a whole batch of knowledge may be extra cost-efficient than calling them row-wise. The instance additionally demonstrates learn how to ahead the invoking consumer’s title from Spark’s TaskContext to the Lambda operate, which may be helpful for attribution:
Get began at the moment
Check out the Public Preview of Enhanced Python UDFs in Unity Catalog – to put in dependencies, to leverage the batched enter mode, or to make use of UC service credentials!
Be part of the UC Compute and Spark product and engineering crew on the Information + AI Summit, June 9–12 on the Moscone Middle in San Francisco! Get a primary take a look at the most recent improvements in information and AI governance and safety. Register now to safe your spot!