This can be a visitor publish by Edijs Drezovs, CEO and Founding father of GOStack, Viesturs Kols, Information Architect at GOStack, and Krisjanis Beitans, Senior Information Engineer at GOStack, in partnership with AWS.
Yggdrasil Gaming develops and publishes on line casino video games globally, processing huge quantities of real-time gaming information for sport efficiency analytics, participant habits insights, and trade intelligence. As Yggdrasil’s system grew, managing dual-cloud environments created operational overhead and restricted their skill to implement superior analytics initiatives. This problem turned important forward of the launch of the Sport in a Field resolution on AWS Market, which generates will increase in information quantity and complexity.
Yggdrasil Gaming decreased multi-cloud complexity and constructed a scalable analytics basis by migrating from Google BigQuery to AWS analytics providers. On this publish, you’ll uncover how Yggdrasil Gaming reworked their information structure to satisfy rising enterprise calls for. You’ll be taught sensible methods for migrating from proprietary techniques to open desk codecs comparable to Apache Iceberg whereas sustaining enterprise continuity.
Yggdrasil labored with GOStack, an AWS Associate, emigrate to an Apache Iceberg-based lakehouse structure. The migration helped cut back operational complexity and enabled real-time gaming analytics and machine studying.
Challenges
Yggdrasil confronted a number of important challenges that prompted their migration to AWS:
- Multi-cloud operational complexity: Managing infrastructure throughout AWS and Google Cloud created important operational overhead, lowering agility and rising upkeep prices. The info workforce needed to keep experience in each environments and coordinate information motion between clouds.
- Structure limitations: The present setup couldn’t successfully assist superior analytics and AI initiatives. Extra critically, the launch of Yggdrasil’s Sport in a Field resolution required a modernized, scalable information surroundings able to dealing with elevated information volumes and enabling superior analytics.
- Scalability constraints: The structure lacked the unified information basis with open requirements and automation required to scale effectively. As information volumes grew, prices elevated proportionally, and the workforce wanted an surroundings designed for contemporary analytics at scale.
Resolution overview
Yggdrasil labored with GOStack, an AWS APN companion, to design their new lakehouse structure. The next diagram reveals the excessive stage overview of this structure.
Yggdrasil efficiently migrated from Google BigQuery to an information lakehouse structure utilizing Amazon Athena, Amazon EMR, Amazon Easy Storage Service (Amazon S3), AWS Glue Information Catalog, AWS Lake Formation, Amazon Elastic Kubernetes Service (Amazon EKS) and AWS Lambda. Their strategic strategy goals to scale back multi-cloud complexity whereas constructing a scalable basis for his or her Sport in a Field resolution and particular AI/ML initiatives like customized sport suggestions and fraud detection.
The mix of Amazon S3, Apache Iceberg, and Amazon Athena allowed Yggdrasil to maneuver away from provisioned, always-on compute fashions. The Amazon Athena pay-per-query pricing fees just for information scanned, eradicating idle compute prices throughout off-peak durations. Inside value modeling carried out throughout the analysis part indicated that this structure may cut back analytics system prices by 30–50% in comparison with compute-based warehouse pricing fashions of different options, significantly for bursty workloads pushed by sport launches, tournaments, and seasonal visitors. By adopting AWS-native analytics providers, Yggdrasil decreased operational complexity by way of native integration with AWS Id and Entry Administration (AWS IAM), Amazon EKS, and AWS Lambda, serving to simplify safety, governance, and automation throughout the analytics system.
The answer facilities on a contemporary lakehouse structure constructed on Amazon S3, which supplies sturdy and cost-efficient storage for Iceberg tables in Apache Parquet format. Apache Iceberg desk format supplies ACID transactions, schema evolution, and time journey capabilities whereas sustaining an open normal. AWS Glue Information Catalog serves because the central technical metadata repository, whereas Amazon Athena acts because the serverless question engine utilized by dbt-athena and for ad-hoc information exploration. Amazon EMR runs Yggdrasil’s legacy Apache Spark utility in a totally managed surroundings, and AWS Lake Formation supplies centralized safety and governance for information lakes, permitting fine-grained entry management at database, desk, column, and row ranges.
The migration adopted a phased strategy:
- Set up lakehouse basis – Arrange Apache Iceberg-based structure with Amazon S3 with AWS Glue Information Catalog
- Implement real-time information ingestion – Deploy Debezium connectors for real-time change information seize from EKS and Google Kubernetes Engine (GKE) clusters
- Migrate processing pipelines – Re-system ETL pipelines utilizing AWS Lambda, and legacy information functions re-systemed on Amazon EMR
- Modernizing the transformation layer – Implement dbt with Amazon Athena for modular, reusable fashions
- Allow governance – Configure AWS Lake Formation for complete information governance
Set up lakehouse basis
The primary part of the migration targeted on constructing a strong basis for the brand new information lakehouse structure on AWS. The purpose was to create a scalable, safe, and cost-efficient surroundings that would assist analytical workloads with open information codecs and serverless question capabilities.
GOStack provisioned an Amazon S3-based information lake because the central storage layer, offering just about limitless scalability and fine-grained value management. This storage-compute separation allows groups to decouple ingestion, transformation, and analytics processes, with every part scaling independently utilizing essentially the most acceptable compute engine.
To determine dataset interoperability and discoverability, the workforce adopted AWS Glue Information Catalog because the unified metadata repository. The catalog shops Iceberg desk definitions and makes schemas accessible throughout providers comparable to Amazon Athena and Apache Spark workloads on Amazon EMR. Most datasets, each batch and streaming, are registered right here, enabling constant metadata visibility throughout the lakehouse.
The info is saved in Apache Iceberg tables on Amazon S3, chosen for its open desk format, ACID transaction assist, and highly effective schema evolution options. Yggdrasil required ACID transactions for constant monetary reporting and fraud detection, schema evolution to accommodate quickly altering gaming information fashions, and time journey queries to align with regulatory audit necessities.
GOStack constructed a customized schema conversion and desk registration service. This inside device converts source-system Avro schemas into Iceberg desk definitions and manages the creation and evolution of raw-layer tables. By controlling schema translation and desk registration immediately, the workforce makes certain that metadata stays in step with the supply techniques and supplies predictable, versioned schema evolution aligned with ingestion wants.
The preliminary setup made the next elements:
- Amazon S3 bucket construction design: Applied a multi-layer format (uncooked, curated, and analytics zones) aligned with information lifecycle greatest practices.
- AWS Glue Information Catalog integration: Outlined database and desk schemas with partitioning methods optimized for Athena efficiency.
- Iceberg configuration: Enabled versioning and metadata retention insurance policies to steadiness storage effectivity and question flexibility.
- Safety and compliance: Configured encryption at relaxation utilizing AWS Key Administration Service (AWS KMS), helped implement entry controls through AWS IAM and Lake Formation, and carried out Amazon S3 bucket insurance policies following the precept of least privilege.
The redesign of the earlier GCP setup helped ship price-performance enhancements. Yggdrasil decreased ingestion and processing prices by roughly 60% whereas additionally decreasing operational overhead by way of a extra direct, event-driven pipeline.
Implement real-time information ingestion
After establishing the lakehouse structure, the following step targeted on enabling real-time information ingestion from Yggdrasil’s operational databases into the uncooked information layer of the lakehouse. The target was to seize and ship transactional modifications as they happen, ensuring that downstream analytics and reporting mirror essentially the most up-to-date data.
To realize this, GOStack deployed Debezium Server Iceberg, an open-source venture that integrates change information seize (CDC) immediately with Apache Iceberg tables. It was deployed as Argo CD functions on Amazon EKS and used Argo’s GitOps-based mannequin for reproducibility, scalability, and seamless rollouts.
This structure supplies an environment friendly ingestion pathway – streaming information modifications immediately from the supply system’s outbox tables into the Apache Iceberg tables registered within the AWS Glue Information Catalog and bodily saved on Amazon S3, bypassing the necessity for intermediate brokers or staging providers. By writing information within the Iceberg desk format, the ingestion layer maintained transactional ensures and quick question availability by way of Amazon Athena.
As a result of Yggdrasil’s supply techniques emitted outbox occasions containing Avro data, the workforce carried out a customized outbox-to-Avro transformation inside Debezium. The outbox desk saved two key elements:
- The Avro schema definition
- The JSON-encoded payload of every file
The customized transformation module mixed these components into legitimate Avro data earlier than persisting them into the goal Iceberg tables. This strategy preserved schema constancy and verified compatibility with downstream processing instruments.
To dynamically route incoming change occasions, the workforce leveraged Debezium’s occasion router configuration. Every file was routed to the suitable Apache Iceberg desk (backed by Amazon S3) based mostly on matter and metadata guidelines, whereas desk schemas and partitioning had been ruled on the AWS Glue facet to keep up stability and alignment with the lakehouse’s information group requirements.
This setup helped ship low-latency ingestion with end-to-end streaming from database outbox to S3-based Iceberg tables in close to actual time. The workforce managed operations finish to finish on Amazon EKS utilizing Helm charts deployed through Argo CD in a GitOps mannequin for absolutely declarative, version-controlled operations. ACID-compliant Iceberg writes verified that partially written information couldn’t corrupt downstream analytics. The modular transformation logic allowed future growth to new supply techniques or occasion codecs with out rearchitecting the ingestion pipeline.
This Debezium Server resolution supplies quick, real-time information ingestion. GOStack considers it an interim structure. In the long run, the ingestion pipeline will evolve to make use of Amazon Managed Streaming for Apache Kafka (Amazon MSK) because the central occasion spine. Debezium connectors will act as producers, publishing change occasions to Apache Kafka subjects, whereas Apache Flink functions will eat, course of, and write information into Iceberg tables.
This deliberate evolution towards a Kafka-based streaming structure verifies Yggdrasil’s lakehouse stays not solely scalable and cost-efficient as we speak, but in addition future-ready – able to supporting richer streaming analytics and broader information integration eventualities because the group grows.
Migrate processing pipelines
As soon as real-time information ingestion was established, GOStack turned its focus to modernizing the info transformation layer. The purpose was to simplify the transformation logic, cut back operational overhead, and unify the orchestration of analytical workloads throughout the new AWS-based lakehouse.
GOStack adopted a lift-and-shift strategy for a few of Yggdrasil’s information pipelines to assist a quick and low-risk transition away from GCP. The light-weight Cloud Run capabilities that beforehand dealt with extraction duties – pulling information from file shares, SharePoint, Google Sheets, and varied third-party APIs – had been re-implemented utilizing AWS Lambda. These Lambda capabilities now combine with the identical exterior techniques and write information immediately into Iceberg tables.
For extra advanced processing, earlier Apache Spark functions operating on Dataproc had been migrated to Amazon EMR with minimal code modifications. This allowed it to protect the prevailing transformation logic whereas benefiting from the managed scaling capabilities of EMR and improved value management on AWS.
Over time, these processes will likely be steadily refactored and consolidated into containerized workflows on the EKS cluster, absolutely orchestrated by Argo Workflows. This phased migration permits Yggdrasil to maneuver workloads to AWS rapidly and decommission GCP assets sooner, whereas nonetheless leaving room for steady enchancment and modernization of the info system over time.
Lastly, a variety of analytical transformations that beforehand lived as BigQuery saved procedures and scheduled queries, that had been now rebuilt as modular dbt fashions executed with dbt-athena. This shift made transformation logic extra clear, maintainable, and version-controlled, enhancing each developer expertise and long-term governance.
Modernizing the transformation layer
With the ingestion pipelines migrated to AWS, GOStack turned its focus to simplifying and modernizing Yggdrasil’s analytical transformations. Slightly than replicating the earlier stored-procedure–pushed strategy, the workforce rebuilt the transformation layer utilizing dbt to assist enhance maintainability, lineage visibility, orchestration, and long-term governance.As a part of this redesign, a number of information fashions had been reshaped to suit the brand new lakehouse structure. Essentially the most important effort concerned rewriting a important Spark-based monetary transformation right into a set of SQL-driven dbt fashions. This shift not solely aligned the logic with the lakehouse design but in addition eliminated the necessity for long-running Spark clusters, serving to generate operational and price financial savings.For the curated information layers, changing the legacy warehouse, GOStack consolidated quite a few scheduled queries and saved procedures into structured dbt fashions. This supplies standardized, version-controlled transformations and clear lineage throughout the analytical stack.
Orchestration was simplified as nicely. Beforehand, coordination was cut up between Apache Airflow for Spark workloads and scheduled queries analytical transformations, creating operational friction and dependency dangers. Within the new structure, Argo Workflows on Amazon EKS orchestrates dbt fashions centrally, consolidating the transformation logic inside a single workflow engine. Whereas most transformations nonetheless run on time-based schedules as we speak, the system now helps event-driven execution by way of Argo Occasions, giving the chance to progressively undertake trigger-based workflows because the transformation layer evolves.
This unified orchestration framework can deliver a number of advantages:
- Consistency: One orchestration layer for information workflows throughout ingestion and transformation.
- Automation: Occasion-driven dbt runs assist take away handbook scheduling and cut back operational overhead.
- Scalability: Argo Workflows scales with the EKS cluster, dealing with concurrent dbt jobs seamlessly.
- Observability: Centralized logging and workflow visualization assist enhance visibility into job dependencies and information freshness.
By way of this transformation, Yggdrasil efficiently unified its information lakes and warehouses into a contemporary lakehouse structure, powered by open information codecs, serverless question engines, and modular transformation logic. The transfer to dbt and Athena not solely simplified operations but in addition helped pave the way in which for quicker iteration, easier governance, and better developer productiveness throughout the info surroundings.
Lakehouse efficiency optimizations
Whereas efficiency tuning is an ongoing journey, as a part of the transformation redesign, GOStack made few performance-oriented tweaks to ensure Athena queries could be quick and cost-efficient. The Apache Iceberg tables had been saved in Parquet with ZSTD compression, offering sturdy learn efficiency and lowering the quantity of information scanned by Athena.
Partitioning methods had been additionally aligned to precise entry patterns utilizing Iceberg’s native partitioning. Uncooked information zones had been partitioned by ingestion timestamp, enabling environment friendly incremental processing. Curated information used business-driven partition keys, comparable to participant or sport identifiers and date dimensions, to assist optimize analytical queries. These designs made certain Athena may prune unneeded information and persistently scan solely the related partitions.
Iceberg’s native partitioning options, together with transforms comparable to bucketing and time slicing, substitute conventional Hive partitioning patterns. As a result of Iceberg manages partitions internally in its metadata layer, not all Glue or Athena partition constructs apply. Counting on Iceberg’s native partitioning helps present predictable pruning and constant efficiency throughout the lakehouse with out introducing legacy Hive behaviors.
To deal with the excessive quantity of small recordsdata produced by real-time ingestion, GOStack enabled AWS Glue Iceberg compaction. This robotically merges small Parquet recordsdata into bigger segments, serving to enhance question efficiency and cut back metadata overhead with out handbook intervention.
Allow governance
The workforce adopted AWS Lake Formation as the first governance layer for the curated zone of the lakehouse, leveraging Lake Formation hybrid entry mode to handle fine-grained permissions alongside present IAM-based entry patterns. This hybrid mode supplies an incremental and versatile pathway to undertake Lake Formation with out forcing a full migration of legacy permissions or inside pipeline roles, making it a really perfect match for Yggdrasil’s phased modernization technique.
Lake Formation provides centralized authorization, supporting database, desk, column, and, critically for Yggdrasil, row-level permissions. These capabilities are important due to the corporate’s multi-tenant working mannequin:
- Sport growth companions require entry to information and studies pertaining solely to their very own video games, facilitating each safety and compliance alignment with companion agreements.
- iGaming operators integrating with Yggdrasil’s system should obtain operational and monetary insights solely for their very own information, enforced robotically by way of reporting instruments backed by curated Iceberg tables.
With Lake Formation hybrid entry mode, tenant-specific row-level entry insurance policies are persistently enforced throughout Amazon Athena, AWS Glue, and Amazon EMR, with out introducing breaking modifications to present IAM-based workloads. This allowed Yggdrasil to implement sturdy governance for exterior customers whereas retaining inside operations secure and predictable.
Internally, Lake Formation can be used to grant the Analytics workforce and BI instruments focused entry to curated datasets, simple however centrally managed to keep up consistency and cut back administrative overhead.
For ingestion and transformation workloads, the workforce continues to depend on IAM roles and insurance policies. Providers comparable to Debezium, dbt, and Argo Workflows require broad however managed entry to uncooked and intermediate storage layers, and IAM supplies a simple, least-privilege mechanism for granting these permissions with out involving Lake Formation within the inside pipeline path.
By adopting Lake Formation in hybrid entry mode and mixing it with IAM for inside providers, Yggdrasil established a governance mannequin that may steadiness sturdy safety with operational flexibility – enabling the lakehouse to scale securely because the enterprise grows.
Outcomes and enterprise influence
The brand new lakehouse, constructed on Amazon Athena, Amazon S3, and AWS Glue Information Catalog, now underpins superior analytics and AI/ML use circumstances comparable to participant habits modeling, predictive sport suggestions, and fraud detection.
The optimized lakehouse design permits Yggdrasil to quickly onboard new analytics workloads and enterprise use circumstances, serving to ship measurable outcomes:
- Diminished operational complexity by way of consolidation on AWS analytics providers
- Value optimization with a 60% discount in information processing prices
- Improved information freshness with 75% decrease latency for analytics outcomes (from 2 hours to half-hour)
- Enhanced governance utilizing the AWS Lake Formation fine-grained controls
- Future-ready structure leveraging open codecs and serverless analytics
Conclusion
Yggdrasil Gaming’s migration journey illustrates how organizations can efficiently transition from proprietary analytics techniques to an open, versatile lakehouse structure. By following a phased strategy guided by AWS Nicely-Architected Framework rules, Yggdrasil maintained enterprise continuity whereas establishing a contemporary basis for his or her information wants.
Primarily based on this expertise, a number of classes emerged to assist information your personal transfer to an AWS-based lakehouse:
- Assess your present state: Establish ache factors in your present information structure and set up clear aims for modernization.
- Begin small: Start with a pilot venture utilizing AWS analytics providers to validate the lakehouse strategy on your particular use circumstances.
- Design for openness: Leverage open desk codecs like Apache Iceberg to keep up flexibility and keep away from vendor lock-in.
- Implement steadily: Observe a phased migration technique just like Yggdrasil’s, prioritizing high-value workloads.
- Optimize constantly: Use efficiency tuning methods for Amazon Athena to assist maximize effectivity and decrease prices.
To be taught extra about constructing fashionable lakehouse architectures, seek advice from “The lakehouse structure of Amazon SageMaker”.
In regards to the authors