When engineering groups at Slack, Reddit, Canva, Airbnb, Shopify, and Uber have to ship code with confidence, they depend on Buildkite. As a CI/CD platform, Buildkite orchestrates complicated construct, take a look at, and deployment pipelines for among the most demanding engineering organizations on the earth. It handles all the things from routine code commits to synthetic intelligence (AI) model-training workloads, processing over 50 billion requests monthly.
On the coronary heart of Buildkite’s take a look at orchestration portfolio is Check Engine, a specialised analytics product designed to assist engineering groups perceive and optimize their take a look at suites at scale. Check Engine aggregates outcomes throughout hundreds of builds, flags flaky assessments, runs parallel take a look at execution throughout machine fleets, and delivers interactive analytics on take a look at execution information. It helps arbitrary metadata tagging for dimensions like occasion kind, structure, language model, cloud supplier, and have flags.
The problem? Delivering all of this in actual time, throughout a number of enterprise tenants, at a quantity that might stress even probably the most strong information infrastructure. On this put up, we discover how Buildkite makes use of Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink to energy Check Engine’s streaming-first analytics structure at scale.
The issue: When scale breaks conventional architectures
Buildkite’s Check Engine should ingest and serve analytics on take a look at telemetry from hundreds of distributed pipelines concurrently, for a number of enterprise prospects. The size is unforgiving: 50 billion take a look at executions monthly, 500K occasions per second at peak ingestion, and webhook payloads reaching 21 MB.
The architectural evolution and its limits
The unique Rails and PostgreSQL stack couldn’t maintain this progress. In 2024, the workforce re-architected round a distributed streaming layer, a stateful stream processor for pre-aggregations, and a number of specialised shops: a key-value retailer for quick lookups, a relational database for pre-computed aggregates, and an open desk format (Iceberg) with a distributed question engine (Trino) for versatile querying.
But the core stress remained unsolved. Enterprise prospects demanded interactive, arbitrary slicing of billions of information throughout high-cardinality dimensions, not canned stories. The stream processor couldn’t deal with advert hoc aggregations at question time. The important thing-value retailer was blind to analytical queries. The distributed question engine provided flexibility however was too gradual for interactive use.
The consequence was a system that was costly and operationally complicated. It included 9 relational database clusters, sprawling ETL pipelines, and 24/7 pre-aggregation jobs operating no matter demand. It nonetheless couldn’t ship the one factor prospects wanted most: quick, versatile, interactive analytics at scale.
Structure and implementation: MSK and Amazon Managed Service for Apache Flink because the streaming spine
The answer Buildkite arrived at facilities on Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink because the real-time information streaming and processing layers, decoupling high-throughput ingestion from downstream analytics.
The information pipeline
The next diagram exhibits the end-to-end information move from CI/CD brokers by means of Amazon MSK and Amazon Managed Service for Apache Flink to the analytics layer.
Amazon MSK sits on the important junction between information producers (the distributed CI/CD brokers and take a look at collectors operating throughout buyer infrastructure) and the downstream processing and analytics layers. Amazon Managed Service for Apache Flink then transforms these uncooked occasion streams into enriched, queryable information earlier than it reaches the analytics retailer.
Excessive-throughput ingestion from CI/CD pipelines
Amazon MSK’s position begins at ingestion. Check collectors embedded in CI/CD pipelines publish take a look at execution occasions on to Kafka matters. The present Amazon MSK cluster handles between 5 MB/sec and 100 MB/sec of inbound information beneath regular working circumstances. The structure is designed to soak up the numerous variance inherent in CI/CD workloads, the place pipeline exercise is bursty and correlated with engineering workforce working hours throughout world time zones.
When the Buildkite challenge was initiated, MSK Specific Brokers weren’t but out there, main the workforce to undertake MSK Tiered Storage as the first mechanism for scaling and restoration. With MSK Specific Brokers now usually out there, the workforce is evaluating a migration of its most important log ingestion workload, which sustains as much as 1 GB/s at peak ingestion. MSK Specific Brokers convey automated storage scaling with zero storage administration overhead, as much as 20x sooner scaling and 90% sooner dealer restoration, 3x increased per-broker throughput, 5x extra partitions per dealer, and built-in Clever Rebalancing.
Actual-time stream processing with Amazon Managed Service for Apache Flink
Sitting between Amazon MSK and the analytics layer, Amazon Managed Service for Apache Flink acts because the stateful stream processing engine that transforms uncooked occasion streams earlier than they attain downstream techniques. Buildkite chosen Flink for its exactly-once processing, mature stateful computation mannequin, and deep Kafka integration. Dealing with sustained peaks of over 25,000 occasions per second, Amazon Managed Service for Apache Flink eliminates the operational overhead of cluster provisioning, model upgrades, checkpointing, and job restoration. This frees engineering groups to give attention to utility logic.
Amazon Managed Service for Apache Flink powers key stateful processing duties, together with flaky take a look at detection by means of time-windowed sample matching, enriching execution occasions with pipeline and buyer metadata, and routing processed information to downstream techniques comparable to ClickHouse for analytics, PostgreSQL for operational workloads, and Amazon Easy Storage Service (Amazon S3) for long-term archival.
Reliability and fault tolerance
Amazon MSK’s three-replica configuration ensures that no single dealer failure could cause information loss or ingestion interruption. Mixed with versatile information retention, the structure supplies a significant replay window. If a downstream shopper (Amazon Managed Service for Apache Flink, ClickHouse, or one other service) experiences an outage, it could resume processing from its final dedicated offset with out information loss.
Throughout the migration to the present structure, Buildkite employed a dual-write technique: concurrently writing to each the present PostgreSQL pipeline and the brand new Amazon MSK/ClickHouse path. This strategy allowed the workforce to validate information consistency and steadily shift visitors with out risking customer-facing disruption. This sample speaks to the operational maturity Amazon MSK supplies.
Operational effectivity positive aspects
The shift to a streaming-first structure, mixed with the downstream simplification of the analytics engine, produced important operational enhancements:
- Flink workloads decreased by 60%+: Eliminating pre-aggregation jobs that ran constantly no matter demand.
- Key/worth retailer utterly retired: Amazon MSK’s buffering functionality, mixed with ClickHouse’s question efficiency, eradicated the necessity for a separate fast-lookup retailer.
- PostgreSQL capability reduce in half: 9 separate database clusters consolidated and right-sized.
- Hundreds of strains of utility code deleted: Easier structure means much less ETL code, fewer failure modes, and sooner onboarding for brand new engineers.
Platform efficiency at a look
| Metric | Worth |
| Month-to-month take a look at executions (for take a look at engine platform) | 50 billion (4x progress from 3B) |
| Sustained peak ingestion | 500K occasions/second |
| Complete information in analytics retailer | 200 billion |
| Log ingestion requests | 70,000+ per second |
| Peak webhook throughput | 1.7 GB/second |
| MSK inbound throughput vary | 5 MB/sec – 100 MB/sec |
Enterprise and developer influence
The technical structure finally exists to serve one goal: serving to builders ship higher software program sooner. The streaming-first structure constructed on Amazon MSK and Amazon Managed Service for Apache Flink delivers on that promise throughout 4 dimensions.
On-demand analytics changed pre-computed stories. Prospects can now interactively slice and cube 70 billion information throughout arbitrary metadata dimensions. They get solutions to queries like “Present me P50 take a look at durations by occasion kind and structure for the final 30 days” in seconds, not hours. Actual-time log streaming by means of the “stay tail” function means builders not watch for a construct to finish earlier than diagnosing failures. At 25,000 occasions per second, this expertise scales throughout hundreds of concurrent enterprise pipelines with out degradation.
Smarter take a look at intelligence comes from Amazon Managed Service for Apache Flink’s stateful flaky take a look at detection: when a take a look at begins exhibiting intermittent failure patterns, Amazon Managed Service for Apache Flink identifies it because it occurs, not after the very fact. That is what separates a proactive analytics platform from a reactive one. It requires publishing information to Kafka, processing with Flink, and letting ClickHouse deal with the complicated learn requests.
Conclusion: Streaming as a strategic basis
Buildkite’s journey from a Rails/Postgres monolith to a streaming-first analytics platform displays a sample more and more frequent amongst enterprise SaaS firms: a dependable, high-throughput streaming and processing layer shouldn’t be an optimization. It’s a prerequisite for working at scale.
Amazon MSK and Amazon Managed Service for Apache Flink kind the spine that helps Buildkite ingest 50 billion take a look at executions monthly, serve real-time interactive analytics to enterprise prospects, and achieve this at decrease value than the extra complicated structure it changed. Amazon MSK handles sturdy, elastic occasion buffering. Amazon Managed Service for Apache Flink transforms uncooked streams into enriched, queryable information. Collectively they take up the operational complexity that might in any other case eat engineering capability.
For platform engineers evaluating streaming infrastructure for multi-tenant SaaS workloads, the sign is obvious: spend money on the streaming spine early, and let managed companies deal with the operational complexity.
To be taught extra about Amazon MSK and Amazon Managed Service for Apache Flink, go to aws.amazon.com/msk and aws.amazon.com/managed-service-apache-flink.
In regards to the authors