Constructing a scalable personalised suggestion system on AWS: From batch to real-time

Amazon.com receives tens of millions of visits on daily basis, and behind each product suggestion on our web site is a system that should course of buyer indicators, run machine studying (ML) fashions, and ship outcomes earlier than the subsequent go to. Doing this throughout international marketplaces for tens of millions of consumers at tens of hundreds of requests per second, whereas protecting experimentation quick and infrastructure prices bounded, is an orchestration problem as a lot as a machine studying one.

Our crew constructed a system that addresses this problem. This put up reveals how we did it utilizing a batch-first structure with AWS Lake Formation, Amazon Managed Workflows for Apache Airflow (Amazon MWAA), Amazon Athena, AWS Glue, Amazon SageMaker, and Amazon DynamoDB, and the way we later prolonged it with Amazon MemoryDB for real-time vector similarity search after we wanted to include extra real-time indicators.

Structure overview

Knowledge flows from the centralized knowledge lake (Lake Formation and Athena) by means of Airflow-orchestrated pipelines utilizing Glue for processing and SageMaker for ML workloads, into DynamoDB for batch serving and MemoryDB for real-time inference

The info lake basis: Centralized entry with Lake Formation

Each suggestion pipeline begins with knowledge. We constructed a centralized knowledge lake to create a single supply of fact that any pipeline or client can entry with out duplicating knowledge or constructing bespoke extract, rework, and cargo (ETL) pipelines.

Golden datasets: Shared as soon as, used in every single place

Earlier than the info lake, every suggestion pipeline independently extracted and reworked its personal copy of product catalog, transaction historical past, and embeddings. This led to delicate inconsistencies: one pipeline may use a barely completely different be part of logic or a stale snapshot, making it troublesome to match mannequin efficiency or debug discrepancies throughout pipelines.

Now, we publish curated, validated datasets as soon as, and each client (Airflow DAGs, ML notebooks, analytics dashboards) reads from the identical tables by means of the identical ruled entry. This implies:

• New pipelines begin quicker. A brand new suggestion mannequin doesn’t want its personal knowledge extraction logic. It queries the present golden datasets from day one.
• Consistency throughout fashions. Once we examine mannequin A to mannequin B, we all know each fashions are skilled and inferred on the identical underlying knowledge.
• Cross-team collaboration. A number of groups share the identical tables as a single supply of fact.
• Two-way knowledge circulate. The info lake serves as each supply and vacation spot. Our pipelines learn golden datasets as inputs and write computed outputs (mannequin scores, function units, intermediate outcomes) again to the lake, the place they change into inputs for different pipelines. This creates a compounding impact: every new pipeline enriches the lake for the subsequent one.

Why Lake Formation?

Our knowledge is saved in Amazon Easy Storage Service (Amazon S3), partitioned by market. Our knowledge customers (Airflow pipelines, ML notebooks, analytics instruments) stay in separate AWS accounts from the info producers. We selected AWS Lake Formation as a result of it provides governance on prime of S3 with out requiring knowledge migration:

• High-quality-grained cross-account entry. Grant table-level permissions per client position, with out managing bucket insurance policies manually.
• Schema governance by means of AWS Glue Knowledge Catalog. Scheduled Glue crawlers infer schemas from information in S3, protecting the catalog present as knowledge evolves.
• Multi-Area consistency. We deploy equivalent infrastructure throughout a number of AWS Areas utilizing AWS Cloud Growth Equipment (AWS CDK), every Area serving its native marketplaces.

Orchestration: Amazon Managed Workflows for Apache Airflow (Amazon MWAA)

We selected Amazon Managed Workflows for Apache Airflow (Amazon MWAA) as our orchestration layer. MWAA removes the operational burden of managing Airflow infrastructure: computerized scaling of staff, built-in excessive availability, and managed upgrades imply our crew focuses on pipeline logic somewhat than cluster upkeep. MWAA lets us outline complicated multi-step workflows with wealthy dependencies in Python code, and its operator mannequin lets us encapsulate team-specific conventions into reusable constructing blocks.

Every suggestion pipeline follows a constant sample:

We constructed a library of reusable customized Airflow operators, every encapsulating one in every of our core compute engines. This diminished new pipeline improvement from weeks to days in our crew’s expertise.

Athena and AWS Glue: Knowledge entry and processing

Amazon Athena is how our pipelines learn from Lake Formation. Our customized operator runs SQL queries towards the Glue Knowledge Catalog and routinely runs UNLOAD to put in writing outcomes to S3: serverless, no infrastructure to handle, and built-in with the Lake Formation permission mannequin.

AWS Glue handles compute-intensive knowledge transformations by means of PySpark: becoming a member of datasets, filtering, deduplication, aggregation, and formatting ML outputs into ultimate suggestion lists. We configure Glue with Auto Scaling employee swimming pools (for instance, 2–50 staff of G.8X kind) so jobs scale with knowledge quantity per market. All jobs run ephemerally: they learn from S3, write to S3, and require no long-running clusters.

Amazon SageMaker: Coaching, inference, and vector search

Amazon SageMaker powers the ML-intensive levels of our pipelines throughout three workload sorts, all orchestrated as steps inside our Airflow DAGs:

Coaching

We use SageMaker Coaching Jobs to coach our suggestion fashions on GPU situations (for instance, ml.g5). Coaching knowledge is ready by upstream Glue jobs and staged in S3. Our customized Airflow operator submits the coaching job, displays its progress, and registers the ensuing mannequin artifact in S3. As soon as coaching completes, the artifact is instantly obtainable for batch inference or endpoint deployment throughout the similar DAG run. This implies a single DAG can go from uncooked knowledge to skilled mannequin to deployed inference with out guide handoffs, and we are able to retrain it on contemporary knowledge each pipeline cycle with zero operator intervention.

Batch inference

SageMaker Batch Rework runs our skilled fashions at scale, producing the outputs that feed into downstream rating and publishing steps. Our batch inference operator handles job submission, polls for completion, and writes the output location to the DAG’s S3 conference so the subsequent Glue step can choose it up routinely. Batch Rework lets us run inference with out provisioning persistent infrastructure, and we are able to scale occasion rely and sort independently for each pipeline based mostly on knowledge quantity.

Vector search

Producing suggestions for tens of millions of consumers requires looking out throughout lots of of hundreds of candidate merchandise per buyer. Precise search at this scale is prohibitively costly, so we use approximate nearest neighbor (ANN) search utilizing FAISS to seek out comparable merchandise effectively, run as SageMaker Processing Jobs.

The workflow:

1. Construct a FAISS index over the candidate catalog.
2. Question the index with per-customer vectors to seek out top-Okay nearest neighbors.
3. Return ranked candidate lists per buyer.

We distribute question vectors throughout the SageMaker Processing fleet utilizing S3-based sharding (ShardedByS3Key). Every occasion receives the complete candidate index however solely a fraction of the question vectors. Each occasion builds an equivalent FAISS index, searches its shard of queries, and writes outcomes to S3. The downstream Glue step merges all shards into the ultimate suggestion lists. This lets us scale horizontally by including situations with out altering any code.

Why SageMaker inside MWAA?

Operating SageMaker jobs as MWAA duties (somewhat than standalone) provides us:

• Finish-to-end lineage. Each mannequin coaching run, inference job, and vector search is tracked as a part of a DAG execution. We are able to hint a suggestion in DynamoDB again to the precise coaching run, knowledge snapshot, and ANN search that produced it.
• Retry and failure dealing with. If a SageMaker job fails (spot occasion preemption, transient capability errors), Airflow retries it routinely with backoff. No guide re-runs.
• Useful resource sequencing. Coaching should end earlier than inference, inference earlier than ANN search. The Airflow dependency mannequin handles this naturally with out polling scripts or step perform state machines.
• Unified monitoring. One Airflow dashboard reveals the well being of all pipelines: Glue ETL, SageMaker coaching, SageMaker inference, and DynamoDB publishing. No context-switching between consoles.

Placing it collectively: A whole pipeline instance

Our suggestion technology pipeline illustrates the complete circulate:

Word that the operators proven (GlueSQLOperator, VectorSearchOperator, and others) are customized inner operators constructed on prime of the Airflow AWS supplier, not open-source libraries. Here’s a simplified model of what this seems to be like in code:

from airflow import DAG
from airflow.fashions.baseoperator import chain
from airflow.utils.task_group import TaskGroup

dag = DAG("recommendation_generator", schedule="0 18 * * 4")  # Weekly
task_groups = []

for market in [...]:  # international marketplaces
    with TaskGroup(group_id=market, dag=dag) as group:

        # Step 1: Extract knowledge from lake
        candidates = GlueSQLOperator(
            task_id="select_candidates",
            tables=[f"{marketplace}.catalog", f"{marketplace}.products"],
            sql="select_candidates.sql", dag=dag)

        customer_history = GlueSQLOperator(
            task_id="select_customer_history",
            tables=[f"{marketplace}.transactions"],
            sql="select_customer_vectors.sql", dag=dag)

        prospects = AthenaSQLOperator(
            task_id="select_customers", database=market,
            question="select_customer_cohort.sql", dag=dag)

        # Step 2: Practice mannequin
        practice = ModelTrainingOperator(
            task_id="train_model",
            training_input={"task_id": customer_history.task_id},
            instance_type="ml.g5", dag=dag)

        # Step 3: Batch inference
        inference = BatchInferenceOperator(
            task_id="batch_inference",
            mannequin={"task_id": practice.task_id},
            input_data={"task_id": candidates.task_id}, dag=dag)

        # Step 4: Vector search through SageMaker Processing Job
        ann_search = VectorSearchOperator(
            task_id="ann_search",
            index_input={"task_id": inference.task_id},
            query_input={"task_id": customer_history.task_id},
            okay=60, instance_count=10, dag=dag)

        # Step 5: Merge and rank with Glue
        merge_and_rank = GlueSparkOperator(
            task_id="merge_and_rank",
            script="merge_results.py", dag=dag)

        # Step 6: Publish to DynamoDB
        publish = DynamoDBPublishOperator(
            task_id="publish_recommendations",
            table_name="Suggestions", dag=dag)

        # Job dependencies
        [candidates, customer_history] >> practice >> inference
        [inference, customers] >> ann_search >> merge_and_rank >> publish

    task_groups.append(group)

chain(*task_groups)  # Execute marketplaces sequentially

Key patterns on this DAG

• Market isolation. Every market runs in its personal TaskGroup. A failure in a single doesn’t block the others.
• Parallel knowledge extraction. Unbiased Athena/Glue queries run concurrently earlier than converging on the coaching step.
• Sequential market execution. chain() runs marketplaces separately to keep away from useful resource rivalry throughout massive SageMaker and Glue jobs.
• Reusable operators. We constructed customized operators like GlueSQLOperator, VectorSearchOperator, and DynamoDBPublishOperator that encode our crew’s conventions (cross-account entry, S3 staging, retry logic, metrics) into shared constructing blocks. New pipelines are largely configuration somewhat than infrastructure code, and these operators are shared throughout dozens of pipelines.
• Implicit knowledge passing. Every operator writes its output to a convention-based S3 path and downstream operators routinely resolve the upstream output location. No hard-coded paths between steps.

This sample powers lots of of pipelines throughout international marketplaces, with every market as an remoted TaskGroup within the DAG.

Serving layer: DynamoDB + ECS

Our Java-based Amazon Elastic Container Service (Amazon ECS) service reads pre-computed suggestions from Amazon DynamoDB at request time:

1. Obtain request with buyer ID, market, buyer context, and web page context.
2. Learn pre-computed suggestions from DynamoDB.
3. Apply real-time filters (availability, eligibility constraints).
4. Re-rank based mostly on real-time context indicators.
5. Return the response.

Amazon DynamoDB is designed to supply single-digit millisecond reads at our scale and time-to-live (TTL) for computerized cleanup of stale suggestions.

Extending to real-time

In suggestion system phrases, our batch pipeline handles the retrieval stage offline. Though this covers nearly all of our visitors, we recognized eventualities the place weekly freshness was not sufficient to seize what prospects are doing proper now. The actual-time extension provides a web based retrieval and rating path for indicators that can’t watch for the subsequent batch cycle.

To deal with these, we prolonged our system with Amazon MemoryDB (which now helps Valkey as its open-source engine) for real-time vector similarity search and SageMaker real-time endpoints for on-demand embedding technology. The identical Airflow pipelines that publish to DynamoDB additionally publish product vectors to MemoryDB (by means of our MemoryDB publish operator), and the identical mannequin in our batch pipeline is deployed to a SageMaker endpoint for single-item inference at request time.

At serving time, after we need to incorporate contemporary indicators, the service calls the SageMaker endpoint to generate an embedding on the fly, then queries MemoryDB for the closest neighbors. These contemporary indicators embody current search queries, cart additions, and different in-session exercise that modifications quicker than our weekly batch cycle. In our workloads, this provides us sub-millisecond vector search latency with out re-running the complete batch pipeline. Critically, the batch pipeline retains the MemoryDB product index contemporary. Our Airflow DAG features a MemoryDB publish operator that refreshes the complete product vector index weekly, so real-time queries all the time search towards an up-to-date index.

Batch and real-time usually are not competing approaches: batch handles slow-moving indicators (buy historical past, catalog relationships) whereas real-time handles fast-moving ones (present session, new arrivals, trending gadgets). Each paths share the identical fashions, the identical knowledge lake, and the identical serving service.

For an in depth deep-dive on this real-time structure, see Actual-time personalised suggestions with Amazon SageMaker and Amazon Managed Valkey.

Safety and entry management

Safety is a foundational concern for a system that spans a number of AWS accounts, processes buyer behavioral knowledge, and runs throughout a number of AWS Areas.

Cross-account entry: Lake Formation grants are issued to consumer-account AWS Id and Entry Administration (IAM) roles, making certain customers can question tables with out direct S3 bucket entry. Every client position receives solely the permissions it wants for its particular tables.

IAM execution roles: Every compute engine (MWAA, Glue, SageMaker) runs below a devoted least-privilege IAM position.

Community isolation: MWAA environments and the ECS serving layer are deployed inside digital non-public clouds (VPCs), with separate VPC configurations per Area.

Reliability and failure dealing with

Regional isolation: Every AWS Area runs an impartial copy of the system. DynamoDB tables are regional, every populated by the native batch pipeline. MemoryDB clusters are regional, with the product vector index refreshed by the native Airflow DAG. This implies a regional failure or pipeline delay in a single Area doesn’t have an effect on different Areas.

Batch pipeline failures: If a pipeline fails mid-run, the earlier DynamoDB knowledge stays stay and continues serving suggestions till the subsequent profitable run. Airflow retries failed duties routinely with configurable backoff. TTL on DynamoDB data bounds how lengthy stale knowledge persists. Failed pipeline runs set off automated alerting so the on-call engineer can examine.

Actual-time path resilience: MemoryDB is deployed in a multi-AZ configuration with computerized failover. The actual-time path is an extension of the batch system, and batch suggestions from DynamoDB stay obtainable no matter real-time path availability.

Classes discovered

1. Begin batch, add real-time incrementally. Batch pipelines are simpler to debug, cheaper to function, and ample for many suggestion eventualities. Add real-time path solely when you’ve clear indication that particular buyer indicators (for instance, in-session exercise, search queries) want sub-hour freshness to stay related.
2. Match the serving path to sign velocity. Not each sign wants real-time processing. Categorize your indicators by how shortly they alter, and route accordingly: batch for sluggish indicators, real-time for quick ones, re-ranking for in-between.
3. Freshness doesn’t all the time require re-computation. A batch-generated candidate set stays largely legitimate between runs. What modifications is relative relevance. Re-ranking at serving time with current buyer exercise gives the look of real-time with out the price of real-time candidate technology.
4. A centralized knowledge lake accelerates every thing. Golden datasets eradicated weeks of per-pipeline knowledge extraction work, made mannequin comparisons reliable, and let new crew members ship their first pipeline in days as a substitute of weeks. The upfront funding in Lake Formation governance paid for itself throughout the first quarter.
5. Put money into reusable operators. Customized Airflow operators encapsulating Athena/Glue/SageMaker patterns let groups ship new pipelines in days. The operators encode finest practices (retry logic, cross-account entry, metrics) so pipeline authors can give attention to enterprise logic.
6. Separate compute from storage. S3 because the common intermediate layer + ephemeral Glue/SageMaker jobs means you pay just for lively computation. No idle clusters between weekly pipeline runs.

Conclusion

We constructed this method as a result of each buyer interplay is a chance to floor the correct product on the proper time. Serving tens of hundreds of requests per second throughout tens of millions of consumers in international marketplaces, our batch-first structure makes use of AWS Lake Formation for ruled knowledge entry, Amazon MWAA for orchestration, Amazon Athena for knowledge lake queries, AWS Glue for distributed processing, Amazon SageMaker for coaching, inference, and vector search, and Amazon DynamoDB for low-latency serving. Once we wanted to include extra real-time indicators, we added Amazon MemoryDB vector search paired with SageMaker real-time endpoints for on-demand embedding technology, extending into real-time with out changing the batch basis.

The structure decisions we’ve made, batch for effectivity and real-time for freshness, all serve the objective of serving to prospects uncover what they want quicker. If you’re constructing personalised experiences at scale, we hope these patterns provide you with a helpful place to begin.

This could not have been potential with out the On a regular basis Necessities engineering crew, whose collective effort turned these concepts right into a manufacturing system serving prospects on daily basis. We’re additionally grateful for the assist and steerage from Sam Heyworth, Nirav Desai, and Ankur Datta, and the broader On a regular basis Necessities management crew.

In regards to the authors