Introducing Apache Spark Join help in AWS Glue interactive classes

Once we constructed AWS Glue interactive classes, our objective was to make AWS Glue as interactive as operating native Python from a pocket book. We principally succeeded. With an easy Python package deal and a Jupyter pocket book, you would execute remotely in opposition to the AWS Glue ephemeral Spark backend. The Livy-based strategy was forward of its time, nevertheless it had limitations from its REST-based protocol. Operating native PySpark unlocked highly effective built-in growth surroundings (IDE) options akin to debugging and linting, so your surroundings may perceive the code and allow you to develop Spark functions extra shortly. Prospects would typically cut up their growth work. They used native Spark (or Docker containers) to develop in an IDE on a small quantity of knowledge, then switched to AWS Glue interactive classes to validate scaling and tuning in opposition to the total dataset.

With trendy PySpark releases got here a brand new protocol: Apache Spark Join. Spark Join bridges the hole between these two worlds: you develop in native Python, however execute on AWS Glue in opposition to precise knowledge. Right this moment, AWS Glue interactive classes help Spark Join natively. You may join from any surroundings that helps the PySpark distant() API, together with VS Code, PyCharm, Amazon SageMaker Unified Studio notebooks, and standalone Python functions. You don’t want to put in specialised kernels or handle cluster infrastructure.

What Spark Join modifications

Spark Join, launched in Spark 3.4, decouples the Spark consumer from the server by means of a light-weight gRPC protocol. As an alternative of operating your driver program on the cluster, your IDE communicates with a distant Spark server by means of a skinny consumer layer. This structure unlocks the important thing workflow enchancment: you develop domestically and execute remotely.

Spark Join structure — skinny consumer with the total energy of Apache Spark

With Spark Join help in AWS Glue interactive classes, you get:

• IDE freedom – Use VS Code, PyCharm, JupyterLab, or any Python surroundings. No kernel set up required.
• Programmatic entry – Construct Spark into your Python functions and automation scripts with a normal SparkSession.builder.distant() name.
• Serverless execution – AWS Glue provisions and manages the Spark cluster. You pay just for the information processing models (DPUs) consumed whereas your session is energetic.
• Spark Join monitoring – The Spark Reside UI now features a devoted Join tab displaying energetic Spark Join classes and operations alongside the prevailing Jobs, Phases, and Executors views.

Getting began with SageMaker Unified Studio

Amazon SageMaker Unified Studio offers essentially the most direct path to Spark Join on AWS Glue. The pocket book surroundings handles session creation, endpoint retrieval, and token refresh robotically, so no connection boilerplate is required.

Prerequisite: You want an Amazon SageMaker Unified Studio undertaking to make use of this workflow. If you happen to don’t have one, create a undertaking in your SageMaker Unified Studio area first.

To hook up with an AWS Glue Spark Join session:

1. Sign up to SageMaker Unified Studio, select your undertaking, and create or open a Pocket book.

A pocket book open in SageMaker Unified Studio

2. Select the compute icon within the left toolbar to open the Compute surroundings panel. Increase the Spark part.

The Compute surroundings panel with the Spark dropdown checklist

3. Choose a Glue Spark connection. Relying in your SageMaker area configuration, you will notice both default.spark or named connections akin to undertaking.spark.compatibility. Choose the suitable Glue (Spark) connection and select Apply.

Linked to Glue Spark Join — operating spark.model returns ‘3.5.6-amzn-1’

After you make your choice, you’re related. The spark session object is offered natively. No imports or configuration are wanted. Begin operating PySpark instantly:

spark.sql("SHOW DATABASES").present()

The session manages itself within the background, together with computerized token refresh.

Utilizing the sagemaker_studio SDK

The sagemaker-studio Python package deal extends the Spark Join expertise past SageMaker Unified Studio notebooks into native IDEs, steady integration and steady supply (CI/CD) pipelines, and any Python surroundings. The sparkutils module handles session initialization and connection configuration in a single name. You get the identical streamlined expertise as within the pocket book, wherever you run Python:

from sagemaker_studio import sparkutils

# Initialize a Glue Spark Join session utilizing your undertaking connection
spark = sparkutils.init(connection_name="default.spark")

# Run queries instantly
spark.sql("SHOW DATABASES").present()

You may as well use sparkutils.get_spark_options() to retrieve pre-configured Java Database Connectivity (JDBC) choices for studying and writing to knowledge sources by means of your undertaking connections. Supported sources embrace Amazon Redshift, Amazon Aurora, and Amazon DocumentDB (with MongoDB compatibility):

# Get connection choices for a Redshift connection in your undertaking
choices = sparkutils.get_spark_options("my_redshift_connection")

# Learn from Redshift through Spark Join
df = spark.learn.format("jdbc").choices(**choices).choice("dbtable", "analytics.orders").load()
df.present()

Inside SageMaker Unified Studio, the sagemaker-studio SDK is native to the surroundings. The spark session and sparkutils can be found with out set up. For native IDE use, set up it with pip set up sagemaker-studio and configure credentials by means of an AWS named profile or boto3 session.

The way it works

Spark Join classes in AWS Glue use a three-step workflow:

1. Create a session – Name the CreateSession API with SessionType set to SPARK_CONNECT. The session provisions in roughly 30 seconds.
2. Retrieve the endpoint – Name GetSessionEndpoint to obtain a sc:// gRPC endpoint URL and a time-limited authentication token.
3. Join with PySpark – Go the endpoint and token to SparkSession.builder.distant() and begin operating Spark operations.

Spark Join protocol move — DataFrame API translated to logical plan, despatched through gRPC/protobuf, outcomes streamed again through gRPC/Arrow

Connecting with the low-level API

Some environments don’t have the sagemaker-studio SDK, akin to customized containers, AWS Lambda features, or non-Python toolchains. In these environments, or should you’re not utilizing SageMaker Unified Studio, you should utilize the AWS SDK (Boto3) to handle classes straight. The next instance demonstrates the total workflow:

import time, boto3, urllib.parse
from pyspark.sql import SparkSession

glue = boto3.consumer("glue", region_name="us-east-1")

# 1. Create a Spark Join session
session_id = "my-spark-connect-session"
glue.create_session(
    Id=session_id,
    Function="arn:aws:iam::123456789012:position/GlueServiceRole",
    Command={"Title": "glueetl"},
    GlueVersion="5.1",
    SessionType="SPARK_CONNECT",
    DefaultArguments={"--enable-spark-live-ui": "true"},
)

# 2. Look ahead to the session to succeed in READY
whereas True:
    standing = glue.get_session(Id=session_id)["Session"]["Status"]
    if standing == "READY":
        break
    time.sleep(5)

# 3. Get the Spark Join endpoint
sc = glue.get_session_endpoint(SessionId=session_id)["SparkConnect"]
endpoint_url = sc["Url"]
auth_token = sc["AuthToken"]

# 4. Join with PySpark
encoded_token = urllib.parse.quote(auth_token, protected="")
connection_string = f"{endpoint_url}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
spark = SparkSession.builder.distant(connection_string).getOrCreate()
spark.sql("SELECT 1 + 1 AS outcome").present()

Monitoring with Spark Reside UI

If you allow the Spark Reside UI at session creation, you acquire entry to a real-time dashboard displaying:

• Jobs and Phases – Observe energetic, accomplished, and failed jobs with stage-level metrics.
• Executors – Monitor reminiscence utilization, shuffle knowledge, and executor well being.
• SQL – Examine question plans and execution particulars.
• Join tab – View energetic Spark Join classes and operations (particular to Spark Join).

Entry the dashboard by means of the GetDashboardUrl API or straight from the AWS Glue console.

import boto3, webbrowser

glue = boto3.consumer("glue", region_name="us-east-1")
dashboard = glue.get_dashboard_url(
    ResourceId="my-spark-connect-session",
    ResourceType="SESSION",
)
webbrowser.open(dashboard["Url"])

In SageMaker Unified Studio, no API name is required. Select Prepared within the pocket book standing bar to open the kernel data popover. From there, open the Spark UI hyperlink for the dwell dashboard or Spark Driver Logs for real-time log output.

Picture displaying “Prepared” within the standing bar to entry Spark UI and Driver Logs straight from the pocket book

Token refresh

Authentication tokens expire after half-hour. In SageMaker Unified Studio, that is dealt with robotically. For programmatic use, you should utilize a background thread to maintain the connection alive. The next helper reconnects transparently earlier than the token expires:

import threading, time, boto3, urllib.parse
from pyspark.sql import SparkSession

class GlueSparkConnect:
    """Maintains a SparkSession with computerized token refresh."""

    def __init__(self, session_id, area="us-east-1", refresh_margin=300):
        self.session_id = session_id
        self.glue = boto3.consumer("glue", region_name=area)
        self.refresh_margin = refresh_margin  # seconds earlier than expiry to refresh
        self._lock = threading.Lock()
        self.spark = self._connect()
        self._start_refresh_loop()

    def _connect(self):
        sc = self.glue.get_session_endpoint(SessionId=self.session_id)["SparkConnect"]
        encoded_token = urllib.parse.quote(sc["AuthToken"], protected="")
        remote_url = f"{sc['Url']}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
        self._token_expiry = sc["AuthTokenExpirationTime"].timestamp()
        return SparkSession.builder.distant(remote_url).getOrCreate()

    def _start_refresh_loop(self):
        def _loop():
            whereas True:
                sleep_for = max(self._token_expiry - time.time() - self.refresh_margin, 30)
                time.sleep(sleep_for)
                with self._lock:
                    self.spark = self._connect()
        t = threading.Thread(goal=_loop, daemon=True)
        t.begin()

# Utilization
session = GlueSparkConnect("my-spark-connect-session")
session.spark.sql("SELECT 1 + 1 AS outcome").present()

The background thread sleeps till 5 minutes earlier than token expiry, then transparently reconnects. As a result of the daemon thread exits when your script ends, there is no such thing as a cleanup required.

Getting began

To start out utilizing Spark Join with AWS Glue interactive classes:

1. Use AWS Glue model 5.1 (Apache Spark 3.5.6).
2. Set up PySpark 3.5.6 domestically: pip set up pyspark==3.5.6.
3. Grant your AWS Id and Entry Administration (IAM) id permissions for glue:CreateSession, glue:GetSession, and glue:GetSessionEndpoint.
4. Create a session with --session-type SPARK_CONNECT and join out of your most popular surroundings.

VPC notice: If you happen to connect with AWS Glue interactive classes by means of a digital non-public cloud (VPC) endpoint, add the brand new Spark Join endpoint (com.amazonaws.{area}.glue.classes) to your VPC configuration. Current AWS Glue VPC endpoints don’t cowl Spark Join site visitors.

For detailed directions, see Connecting to a Spark Join session within the AWS Glue Developer Information.

In regards to the authors