Serverless analytics pipelines utilizing the Apache Spark engine in Amazon Athena

Constructing and sustaining clusters for information processing with Apache Spark has lengthy been a ache level for organizations of all sizes. Conventional deployments require important operational overhead and current a number of challenges that decelerate time-to-insight and enhance complete price of possession. On this put up, we are going to show three integration patterns that allow information groups concentrate on analytics as an alternative of infrastructure administration.

Contemplate the standard expertise of knowledge groups working with self-managed Spark clusters:

• Infrastructure complexity – Groups should handle Amazon Elastic Compute Cloud (Amazon EC2) cases, networking, safety teams, and cluster configurations throughout improvement, staging, and manufacturing environments.
• Value unpredictability – Idle clusters proceed consuming assets and producing payments, whereas automated scaling insurance policies typically lag behind precise demand patterns.
• Operational burden – DevOps groups spend important time patching, monitoring, and troubleshooting cluster well being points.
• Growth friction – Information scientists and engineers should await cluster provisioning earlier than they will start exploratory evaluation, slowing down iterative improvement cycles.
• Interactive workload challenges – Managing interactive Spark workloads usually requires extra parts, exposing particular ports, and complicated community configurations.

These challenges turn into particularly pronounced when organizations have to help a number of concurrent workloads: notebooks for information scientists, scheduled pipelines for information engineers, and advert hoc queries for analysts. The normal strategy encourages groups to decide on between sustaining a number of clusters (costly) or sharing assets (contentious) whereas sustaining fastened endpoint connectivity for interactive workloads (often exposing JDBC ports for the Thrift protocol).

The Apache Spark engine in Amazon Athena addresses these operational challenges by offering a completely managed, serverless Spark execution surroundings. Constructed on Firecracker micro-VMs (AWS’s light-weight virtualization expertise) and operating the AWS-optimized Spark 3.5.6 engine with Spark Join help, Athena with Apache Spark launches and scales in seconds, lowering prices for unpredictable workloads and infrastructure operational overhead.

Athena with Apache Spark is already built-in as a compute engine inside Amazon SageMaker Unified Studio notebooks, offering speedy startup and scaling, making it perfect for advert hoc information exploration and transformations.

This put up reveals how builders, information engineers, and analysts can connect with a safe Spark Join endpoint in Athena with Apache Spark. You should utilize your most popular instruments, akin to Jupyter notebooks, VS Code, or dbt with Apache Airflow, with out managing cluster lifecycle or scaling.

Answer overview

We discover three integration patterns that show how the pliability of Athena with Apache Spark can cut back operational overhead and speed up innovation with on-demand useful resource readiness:

• Sample A: Interactive evaluation with Jupyter notebooks – Information scientists join notebooks on to Athena with Apache Spark for exploratory evaluation and have engineering.
• Sample B: Native improvement with VS Code – Software program engineers develop Spark purposes of their most popular IDE (built-in improvement surroundings) whereas executing on serverless compute.
• Sample C: Scheduled pipelines with dbt + Apache Airflow – Information engineers run manufacturing transformation pipelines with correct orchestration and session lifecycle administration.

The next diagram illustrates the high-level structure for connecting to Athena with Apache Spark utilizing Spark Join.

What’s new within the Apache Spark engine in Amazon Athena

In November 2025, the Apache Spark engine in Amazon Athena launched a major replace with speedy session creation instances and capabilities that weren’t attainable with earlier iterations:

• Safe Spark Join – Provides Spark Join as a completely managed, authenticated, and licensed AWS endpoint for distant connectivity from Spark-compatible instruments. For extra info, see Spark Join help.
• Session-level price attribution – Observe prices per interactive session in AWS Value Explorer or Value and Utilization Studies for granular chargeback and budgeting. For extra info, see Session degree price attribution.
• Superior debugging capabilities – Stay Spark UI and Spark Historical past Server help for debugging workloads from each APIs and notebooks. For extra info, see Accessing the Spark UI.
• AWS Lake Formation integration – Entry AWS Glue Information Catalog tables secured by AWS Lake Formation. For extra info, see Utilizing Lake Formation with Athena for Spark workgroups.

Conditions

To implement this answer, you want the next:

• An AWS account with permissions for Amazon Athena, Amazon Easy Storage Service (Amazon S3), and AWS Glue.
• An Athena with Apache Spark workgroup configured with the newest Spark 3.5.6 engine.
• Python 3.9+ put in domestically.
• AWS credentials configured.

Be aware: This tutorial creates AWS assets that incur prices, together with Athena periods (charged per DPU-hour), Amazon S3 storage, and information switch. Athena periods are charged whereas lively, even when idle throughout the timeout interval. Comply with the cleanup directions on the finish of this put up to keep away from ongoing prices.

Provisioning workflow overview

The workflow for utilizing the Apache Spark engine in Amazon Athena with Spark Join follows these steps:

1. Create the session – Use the AWS API (start_session) to initialize a Spark session. The Spark driver is instantly able to course of requests (no JVM startup time).
2. Get the Spark Join endpoint – Retrieve the endpoint URL and authentication token utilizing get_session_endpoint.
3. Configure Your Instruments – Set the SPARK_REMOTE surroundings variable or configure your software with the Spark Join URL.
4. Run Processing Steps – Run your Spark code as you usually would, however in a completely serverless surroundings that scales robotically primarily based in your wants.
5. Monitor by way of Spark UI – Entry the reside Spark UI for debugging and efficiency monitoring utilizing get_resource_dashboard.
6. Terminate the session – Clear up assets when completed utilizing terminate_session.

By default, the session is configured with autoscaling utilizing Spark Dynamic Useful resource Allocation as much as 60 employees and an idle timeout of 20 minutes. You’ll be able to change the default configuration on the workgroup degree when creating it (create_work_group API) or when creating the session (start_session API).

Sample A: Interactive evaluation with Jupyter notebooks

The Jupyter pocket book integration gives an interactive surroundings for exploratory information evaluation, characteristic engineering, and mannequin preparation. Notebooks join on to Athena with Apache Spark periods for speedy iteration with out cluster administration.

Arrange the surroundings

Create and activate a Python digital surroundings, then set up the required dependencies and begin JupyterLab:

python -m venv athena
supply ./athena/bin/activate
pip set up jupyterlab
pip set up "pyspark[connect]==3.5.6"
pip set up boto3
python -m jupyterlab

Create an Athena with Apache Spark workgroup

Earlier than connecting, create an Athena with Apache Spark workgroup on the AWS Administration Console:

1. Navigate to Amazon Athena → Workgroups → Create workgroup.
2. Choose Apache Spark because the analytics engine.
3. Select the Spark 3.5.6 engine model.
4. Configure the IAM position for the workgroup.
5. Configure the Amazon S3 output location.

Be aware: If you happen to used Athena with Apache Spark beforehand, you could create a brand new workgroup to make use of the newest model with Spark Join help.

Create a session and join

In your Jupyter pocket book, use boto3 to create a session and set up the Spark Join connection:

import boto3

# Initialize the Athena shopper
shopper = boto3.shopper('athena', region_name="us-east-1") # Substitute along with your area

# Begin a brand new Spark session
response=shopper.start_session(
    WorkGroup='your-workgroup-name',
    EngineConfiguration={}
)
session_id=response['SessionId']
print(f"Session created: {session_id}")

# Get the session endpoint and authentication token
response=shopper.get_session_endpoint(SessionId=session_id)
authtoken=response['AuthToken']
endpoint_url=response['EndpointUrl']

# Construct the Spark Join URL
endpoint_url=endpoint_url.substitute("https", "sc") + ":443/;use_ssl=true;"
url_with_headers=f"{endpoint_url}x-aws-proxy-auth={authtoken}"

# Create the Spark session
from pyspark.sql import SparkSession
from pyspark.sql.features import col, rand, sum, avg, depend

spark = SparkSession.builder 
    .distant(url_with_headers) 
    .getOrCreate()

# Confirm the connection
spark.sql("SELECT 1").present()

Run queries and observe automated scaling

Generate a bigger dataset to set off executor scaling. You’ll be able to monitor the scaling conduct by way of the Spark UI:

# Generate massive dataset to set off executor scaling
large_data = spark.vary(0, 10000000, numPartitions=100)

# Heavy computation that can require extra executors
outcome=large_data.choose(
    col("id"),
    (col("id") * col("id")).alias("squared"),
    rand().alias("random")
).groupBy((col("id") % 1000).alias("group")).agg(
    sum("squared").alias("sum_squared"),
    avg("random").alias("avg_random"),
    depend("*").alias("depend")
).orderBy("group")

outcome.present()

Entry the Spark UI

Every session comes with a safe URL serving the Spark UI, to watch and debug purposes:

import os

# Get account ID
sts=boto3.shopper("sts")
account_id=sts.get_caller_identity()["Account"]

# Construct session ARN
partition=os.environ.get("AWS_PARTITION", "aws")
area="us-east-1"
workgroup="your-workgroup-name"
session_arn=f"arn:{partition}:athena:{area}:{account_id}:workgroup/{workgroup}/session/{session_id}"

# Get Spark UI URL
ui_response=shopper.get_resource_dashboard(ResourceARN=session_arn)
print(f"Spark UI: {ui_response['Url']}")

Sample B: Native improvement with VS Code

VS Code integration helps you to develop Spark purposes domestically in your most popular IDE whereas executing on Amazon Athena with Apache Spark compute. This sample is right for constructing reusable libraries, testing transformations, and creating production-ready code.

Arrange the surroundings

Create a digital surroundings and set up dependencies:

python -m venv athena-vscode
supply ./athena-vscode/bin/activate
pip set up "pyspark[connect]==3.5.6"
pip set up boto3

Join from VS Code

The workflow is an identical to Sample A. You begin a session with boto3, construct the Spark Join URL, and create a SparkSession. The important thing distinction is setting the SPARK_REMOTE surroundings variable, which permits SparkSession.builder.getOrCreate() to attach robotically:

import os
import boto3

# Begin session and get endpoint (identical as Sample A)
shopper=boto3.shopper('athena', region_name="us-east-1")
response=shopper.start_session(WorkGroup='your-workgroup', EngineConfiguration={})
session_id=response['SessionId']
response=shopper.get_session_endpoint(SessionId=session_id)
endpoint_url=response['EndpointUrl'].substitute("https", "sc") + ":443/;use_ssl=true;"
spark_remote=f"{endpoint_url}x-aws-proxy-auth={response['AuthToken']}"

# Set surroundings variable for automated connection
os.environ["SPARK_REMOTE"]=spark_remote

# Now SparkSession connects robotically
from pyspark.sql import SparkSession
spark=SparkSession.builder.getOrCreate()

Be aware: The SPARK_REMOTE URL comprises a short-lived authentication token that expires with the session. For manufacturing workloads, retrieve the token on demand utilizing get_session_endpoint() fairly than storing it persistently. Keep away from logging or persisting this worth.

This identical sample works with most Spark-compatible improvement environments. AI coding assistants like Claude Code, Cursor, and Kiro profit notably effectively from this strategy. The flexibility to spin up a recent Athena with Apache Spark session in seconds means builders can quickly iterate on generated code and check transformations instantly. They’ll tear down periods when finished, with out sustaining a persistent cluster between coding periods.

Sample C: Scheduled pipelines with dbt + Airflow

For manufacturing information pipelines, combining dbt (information construct software) with Apache Airflow orchestration gives a sturdy, version-controlled strategy to managing advanced transformation workflows. Athena with Apache Spark executes the dbt fashions with serverless compute, eliminating cluster administration overhead.

Set up dependencies

The important thing dependencies for dbt with Athena with Apache Spark should be put in within the right order:

pip set up pyspark[connect]==3.5.6 # Set up first to make sure right model
pip set up dbt-spark[session]
pip set up setuptools

Essential: Set up pyspark[connect]==3.5.6 first to ensure dbt makes use of the appropriate PySpark model.

Configure dbt profile

Configure dbt to make use of Spark Join with a session-based connection. Create a profiles.yml file:

The technique: session configuration makes use of a neighborhood Spark session. When pyspark[connect]==3.5.6 is put in and the SPARK_REMOTE surroundings variable is about, dbt robotically connects by way of Spark Join.

spark_connect_profile:
  goal: dev
  outputs:
    dev:
      kind: spark
      technique: session
      schema: default
      database: default
      host: NA # Ignored by technique=session
      consumer: dummy # Placeholder
      connect_timeout: 30
      connect_retries: 0

Create a dbt mannequin

Create a dbt mannequin that writes to Apache Iceberg format (fashions/bucketed_data.sql):

{{ config(
    materialized='desk',
    file_format="iceberg",
    catalog='iceberg',
    location_root="s3://your-bucket/iceberg-tables"
) }}

WITH numbers AS (
    SELECT id
    FROM vary(0, 100000)
),
buckets AS (
    SELECT
        id,
        id % 10 AS bucket,
        current_timestamp() AS created_at
    FROM numbers
)
SELECT * FROM buckets

Combine with Airflow

For manufacturing deployments, combine with Apache Airflow (or Amazon Managed Workflows for Apache Airflow (Amazon MWAA)) to orchestrate dbt runs with correct session lifecycle administration.

The DAG follows this sample:

1. setup_athena_session – A PythonOperator that begins the session and pushes spark_remote_url to XCom.
2. run_dbt – A BashOperator that units SPARK_REMOTE from XCom and runs dbt.
3. terminate_athena_session – A PythonOperator with trigger_rule=ALL_DONE to ensure cleanup runs even on failure.

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
from airflow.utils.trigger_rule import TriggerRule
from datetime import datetime

with DAG(
    dag_id="athena_dbt_pipeline",
    schedule="@every day",
    catchup=False,
    start_date=datetime(2025, 1, 1),
) as dag:

    setup_session=PythonOperator(
        task_id="setup_athena_session",
        python_callable=setup_athena_session, # comparable boto3 stream demonstrated earlier
    )

    run_dbt=BashOperator(
        task_id="run_dbt",
        bash_command="""
        export SPARK_REMOTE="{{ (ti.xcom_pull(task_ids="setup_athena_session") or {}).get('spark_remote_url', '') }}"
        supply /path/to/dbt-env/bin/activate
        dbt run --project-dir . --profiles-dir .
        """
    )

    close_session=PythonOperator(
        task_id="terminate_athena_session",
        python_callable=terminate_athena_session,
        trigger_rule=TriggerRule.ALL_DONE,
    )

    setup_session >> run_dbt >> close_session

Safety and finest practices

While you connect with Athena with Apache Spark, observe these practices to guard your information and credentials.

Spark Join safety

Athena with Apache Spark makes use of Spark Hook up with securely transmit queries and obtain outcomes. All communication is encrypted end-to-end utilizing TLS 1.2+. Session tokens are short-lived and robotically rotated.

Suggestions:

• Use IAM roles for authentication fairly than long-lived credentials.
• Session tokens have a restricted lifetime, so refresh them for long-running operations.
• Monitor Spark Join exercise in AWS CloudTrail for audit compliance.

IAM permissions

Implement least-privilege IAM insurance policies. At minimal, the next permissions are required:

• athena:StartSession, athena:TerminateSession, athena:GetSession, athena:GetSessionEndpoint, and athena:GetResourceDashboard in your workgroup.
• Amazon S3 permissions in your information buckets.
• AWS Glue Information Catalog permissions in your database and desk entry.

Clear up

To keep away from ongoing prices, take away the assets created throughout this walkthrough:

1. Terminate any lively Athena periods:

   aws athena terminate-session --session-id 

2. Delete the Athena workgroup you created for this tutorial utilizing the Amazon Athena console or the DeleteWorkGroup API.
3. Take away Amazon S3 objects created throughout testing, together with question outcomes and Iceberg desk information at your configured output location. Information written to Amazon S3 persists after session termination and continues to incur storage prices.
4. Delete any IAM roles created particularly for this walkthrough.
5. Take away any AWS Glue Information Catalog databases and tables created throughout testing.

Conclusion

The Apache Spark engine in Amazon Athena with Spark Join help transforms how groups construct and function Spark workloads. By eliminating cluster administration overhead and offering near-instant, serverless compute, information groups can concentrate on delivering insights fairly than managing infrastructure.

The three patterns lined on this put up show the pliability of Athena with Apache Spark:

• Sample A (Jupyter notebooks) – Best for information scientists doing exploratory evaluation and have engineering.
• Sample B (VS Code) – Nicely-suited for software program engineers constructing production-ready Spark purposes.
• Sample C (dbt + Airflow) – Nicely-suited for information engineers operating scheduled, version-controlled transformation pipelines.

With speedy session creation, automated scaling, and pay-per-use pricing, Athena with Apache Spark gives a compelling various to self-managed Spark clusters.

Further assets

In regards to the authors