Big Data

Migrate from Customary brokers to Categorical brokers in Amazon MSK utilizing Amazon MSK Replicator

fevereiro 14, 2025
techhdesign


Amazon Managed Streaming for Apache Kafka (Amazon MSK) now provides a brand new dealer sort known as Categorical brokers. It’s designed to ship as much as 3 occasions extra throughput per dealer, scale as much as 20 occasions quicker, and cut back restoration time by 90% in comparison with Customary brokers operating Apache Kafka. Categorical brokers come preconfigured with Kafka finest practices by default, help Kafka APIs, and supply the identical low latency efficiency that Amazon MSK clients count on, so you’ll be able to proceed utilizing current consumer functions with none modifications. Categorical brokers present easy operations with hands-free storage administration by providing limitless storage with out pre-provisioning, eliminating disk-related bottlenecks. To study extra about Categorical brokers, confer with Introducing Categorical brokers for Amazon MSK to ship excessive throughput and quicker scaling in your Kafka clusters.

Creating a brand new cluster with Categorical brokers is easy, as described in Amazon MSK Categorical brokers. Nonetheless, in case you have an current MSK cluster, you could migrate to a brand new Categorical based mostly cluster. On this publish, we focus on how you need to plan and carry out the migration to Categorical brokers in your current MSK workloads on Customary brokers. Categorical brokers supply a distinct person expertise and a distinct shared accountability boundary, so utilizing them on an current cluster will not be attainable. Nonetheless, you need to use Amazon MSK Replicator to repeat all information and metadata out of your current MSK cluster to a brand new cluster comprising of Categorical brokers.

MSK Replicator provides a built-in replication functionality to seamlessly replicate information from one cluster to a different. It robotically scales the underlying assets, so you’ll be able to replicate information on demand with out having to observe or scale capability. MSK Replicator additionally replicates Kafka metadata, together with subject configurations, entry management lists (ACLs), and client group offsets.

Within the following sections, we focus on use MSK Replicator to copy the information from a Customary dealer MSK cluster to an Categorical dealer MSK cluster and the steps concerned in migrating the consumer functions from the outdated cluster to the brand new cluster.

Planning your migration

Migrating from Customary brokers to Categorical brokers requires thorough planning and cautious consideration of varied components. On this part, we focus on key facets to deal with through the planning section.

Assessing the supply cluster’s infrastructure and wishes

It’s essential to judge the capability and well being of the present (supply) cluster to verify it may possibly deal with further consumption throughout migration, as a result of MSK Replicator will retrieve information from the supply cluster. Key checks embrace:

    • CPU utilization – The mixed CPU Person and CPU System utilization per dealer ought to stay under 60%.
    • Community throughput – The cluster-to-cluster replication course of provides additional egress visitors, as a result of it’d want to copy the prevailing information based mostly on enterprise necessities together with the incoming information. As an example, if the ingress quantity is X GB/day and information is retained within the cluster for two days, replicating the information from the earliest offset would trigger the whole egress quantity for replication to be 2X GB. The cluster should accommodate this elevated egress quantity.

Let’s take an instance the place in your current supply cluster you’ve a mean information ingress of 100 MBps and peak information ingress of 400 MBps with retention of 48 hours. Let’s assume you’ve one client of the information you produce to your Kafka cluster, which signifies that your egress visitors will likely be similar in comparison with your ingress visitors. Based mostly on this requirement, you need to use the Amazon MSK sizing information to calculate the dealer capability you could safely deal with this workload. Within the spreadsheet, you have to to supply your common and most ingress/egress visitors within the cells, as proven within the following screenshot.

As a result of you could replicate all the information produced in your Kafka cluster, the consumption will likely be greater than the common workload. Taking this under consideration, your total egress visitors will likely be at the least twice the scale of your ingress visitors.
Nonetheless, if you run a replication software, the ensuing egress visitors will likely be greater than twice the ingress since you additionally want to copy the prevailing information together with the brand new incoming information within the cluster. Within the previous instance, you’ve a mean ingress of 100 MBps and you keep information for 48 hours, which suggests that you’ve a complete of roughly 18 TB of current information in your supply cluster that must be copied over on high of the brand new information that’s coming by means of. Let’s additional assume that your objective for the replicator is to catch up in 30 hours. On this case, your replicator wants to repeat information at 260 MBps (100 MBps for ingress visitors + 160 MBps (18 TB/30 hours) for current information) to catch up in 30 hours. The next determine illustrates this course of.

Due to this fact, within the sizing information’s egress cells, you could add an extra 260 MBps to your common information out and peak information out to estimate the scale of the cluster you need to provision to finish the replication safely and on time.

Replication instruments act as a client to the supply cluster, so there’s a likelihood that this replication client can eat greater bandwidth, which might negatively affect the prevailing software consumer’s produce and eat requests. To manage the replication client throughput, you need to use a consumer-side Kafka quota within the supply cluster to restrict the replicator throughput. This makes positive that the replicator client will throttle when it goes past the restrict, thereby safeguarding the opposite shoppers. Nonetheless, if the quota is about too low, the replication throughput will undergo and the replication would possibly by no means finish. Based mostly on the previous instance, you’ll be able to set a quota for the replicator to be at the least 260 MBps, in any other case the replication won’t end in 30 hours.

  • Quantity throughput – Knowledge replication would possibly contain studying from the earliest offset (based mostly on enterprise requirement), impacting your main storage quantity, which on this case is Amazon Elastic Block Retailer (Amazon EBS). The VolumeReadBytes and VolumeWriteBytes metrics ought to be checked to verify the supply cluster quantity throughput has further bandwidth to deal with any further learn from the disk. Relying on the cluster dimension and replication information quantity, you need to provision storage throughput within the cluster. With provisioned storage throughput, you’ll be able to improve the Amazon EBS throughput as much as 1000 MBps relying on the dealer dimension. The utmost quantity throughput could be specified relying on dealer dimension and sort, as talked about in Handle storage throughput for Customary brokers in a Amazon MSK cluster. Based mostly on the previous instance, the replicator will begin studying from the disk and the amount throughput of 260 MBps will likely be shared throughout all of the brokers. Nonetheless, current shoppers can lag, which can trigger studying from the disk, thereby growing the storage learn throughput. Additionally, there’s storage write throughput because of incoming information from the producer. On this state of affairs, enabling provisioned storage throughput will improve the general EBS quantity throughput (learn + write) in order that current producer and client efficiency doesn’t get impacted as a result of replicator studying information from EBS volumes.
  • Balanced partitions – Be sure that partitions are well-distributed throughout brokers, with no skewed chief partitions.

Relying on the evaluation, you would possibly must vertically scale up or horizontally scale out the supply cluster earlier than migration.

Assessing the goal cluster’s infrastructure and wishes

Use the identical sizing software to estimate the scale of your Categorical dealer cluster. Usually, fewer Categorical brokers is likely to be wanted in comparison with Customary brokers for a similar workload as a result of relying on the occasion dimension, Categorical brokers permit as much as thrice extra ingress throughput.

Configuring Categorical Brokers

Categorical brokers make use of opinionated and optimized Kafka configurations, so it’s essential to distinguish between configurations which are read-only and people which are learn/write throughout planning. Learn/write broker-level configurations ought to be configured individually as a pre-migration step within the goal cluster. Though MSK Replicator will replicate most topic-level configurations, sure topic-level configurations are at all times set to default values in an Categorical cluster: replication-factor, min.insync.replicas, and unclean.chief.election.allow. If the default values differ from the supply cluster, these configurations will likely be overridden.

As a part of the metadata, MSK Replicator additionally copies sure ACL varieties, as talked about in Metadata replication. It doesn’t explicitly copy the write ACLs besides the deny ones. Due to this fact, if you happen to’re utilizing SASL/SCRAM or mTLS authentication with ACLs relatively than AWS Identification and Entry Administration (IAM) authentication, write ACLs have to be explicitly created within the goal cluster.

Shopper connectivity to the goal cluster

Deployment of the goal cluster can happen throughout the similar digital non-public cloud (VPC) or a distinct one. Think about any modifications to consumer connectivity, together with updates to safety teams and IAM insurance policies, through the planning section.

Migration technique: Abruptly vs. wave

Two migration methods could be adopted:

  • Abruptly – All subjects are replicated to the goal cluster concurrently, and all shoppers are migrated without delay. Though this strategy simplifies the method, it generates vital egress visitors and includes dangers to a number of shoppers if points come up. Nonetheless, if there’s any failure, you’ll be able to roll again by redirecting the shoppers to make use of the supply cluster. It’s really helpful to carry out the cutover throughout non-business hours and talk with stakeholders beforehand.
  • Wave – Migration is damaged into phases, transferring a subset of shoppers (based mostly on enterprise necessities) in every wave. After every section, the goal cluster’s efficiency could be evaluated earlier than continuing. This reduces dangers and builds confidence within the migration however requires meticulous planning, particularly for big clusters with many microservices.

Every technique has its execs and cons. Select the one which aligns finest with your online business wants. For insights, confer with Goldman Sachs’ migration technique to maneuver from on-premises Kafka to Amazon MSK.

Cutover plan

Though MSK Replicator facilitates seamless information replication with minimal downtime, it’s important to plan a transparent cutover plan. This consists of coordinating with stakeholders, stopping producers and shoppers within the supply cluster, and restarting them within the goal cluster. If a failure happens, you’ll be able to roll again by redirecting the shoppers to make use of the supply cluster.

Schema registry

When migrating from a Customary dealer to an Categorical dealer cluster, schema registry concerns stay unaffected. Shoppers can proceed utilizing current schemas for each producing and consuming information with Amazon MSK.

Answer overview

On this setup, two Amazon MSK provisioned clusters are deployed: one with Customary brokers (supply) and the opposite with Categorical brokers (goal). Each clusters are situated in the identical AWS Area and VPC, with IAM authentication enabled. MSK Replicator is used to copy subjects, information, and configurations from the supply cluster to the goal cluster. The replicator is configured to keep up an identical subject names throughout each clusters, offering seamless replication with out requiring client-side modifications.

In the course of the first section, the supply MSK cluster handles consumer requests. Producers write to the clickstream subject within the supply cluster, and a client group with the group ID clickstream-consumer reads from the identical subject. The next diagram illustrates this structure.

When information replication to the goal MSK cluster is full, we have to consider the well being of the goal cluster. After confirming the cluster is wholesome, we have to migrate the shoppers in a managed method. First, we have to cease the producers, reconfigure them to put in writing to the goal cluster, after which restart them. Then, we have to cease the shoppers after they’ve processed all remaining information within the supply cluster, reconfigure them to learn from the goal cluster, and restart them. The next diagram illustrates the brand new structure.

Migrate from Customary brokers to Categorical brokers in Amazon MSK utilizing Amazon MSK Replicator

After verifying that each one shoppers are functioning appropriately with the goal cluster utilizing Categorical brokers, we are able to safely decommission the supply MSK cluster with Customary brokers and the MSK Replicator.

Deployment Steps

On this part, we focus on the step-by-step course of to copy information from an MSK Customary dealer cluster to an Categorical dealer cluster utilizing MSK Replicator and in addition the consumer migration technique. For the aim of the weblog, “suddenly” migration technique is used.

Provision the MSK cluster

Obtain the AWS CloudFormation template to provision the MSK cluster. Deploy the next in us-east-1 with stack identify as migration.

This may create the VPC, subnets, and two Amazon MSK provisioned clusters: one with Customary brokers (supply) and one other with Categorical brokers (goal) throughout the VPC configured with IAM authentication. It’ll additionally create a Kafka consumer Amazon Elastic Compute Cloud (Amazon EC2) occasion the place from we are able to use the Kafka command line to create and think about Kafka subjects and produce and eat messages to and from the subject.

Configure the MSK consumer

On the Amazon EC2 console, hook up with the EC2 occasion named migration-KafkaClientInstance1 utilizing Session Supervisor, a functionality of AWS Methods Supervisor.

After you log in, you could configure the supply MSK cluster bootstrap handle to create a subject and publish information to the cluster. You may get the bootstrap handle for IAM authentication from the main points web page for the MSK cluster (migration-standard-broker-src-cluster) on the Amazon MSK console, beneath View Shopper Data. You additionally must replace the producer.properties and client.properties information to replicate the bootstrap handle of the usual dealer cluster.

sudo su - ec2-user

export BS_SRC=<>
sed -i "s/BOOTSTRAP_SERVERS_CONFIG=/BOOTSTRAP_SERVERS_CONFIG=${BS_SRC}/g" producer.properties 
sed -i "s/bootstrap.servers=/bootstrap.servers=${BS_SRC}/g" client.properties

Create a subject

Create a clickstream subject utilizing the next instructions:

/residence/ec2-user/kafka/bin/kafka-topics.sh --bootstrap-server=$BS_SRC 
--create --replication-factor 3 --partitions 3 
--topic clickstream 
--command-config=/residence/ec2-user/kafka/config/client_iam.properties

Produce and eat messages to and from the subject

Run the clickstream producer to generate occasions within the clickstream subject:

cd /residence/ec2-user/clickstream-producer-for-apache-kafka/

java -jar goal/KafkaClickstreamClient-1.0-SNAPSHOT.jar -t clickstream 
-pfp /residence/ec2-user/producer.properties -nt 8 -rf 3600 -iam 
-gsr -gsrr <> -grn default-registry -gar

Open one other Session Supervisor occasion and from that shell, run the clickstream client to eat from the subject:

cd /residence/ec2-user/clickstream-consumer-for-apache-kafka/

java -jar goal/KafkaClickstreamConsumer-1.0-SNAPSHOT.jar -t clickstream 
-pfp /residence/ec2-user/client.properties -nt 3 -rf 3600 -iam 
-gsr -gsrr <> -grn default-registry

Maintain the producer and client operating. If not interrupted, the producer and client will run for 60 minutes earlier than it exits. The -rf parameter controls how lengthy the producer and client will run.

Create an MSK replicator

To create an MSK replicator, full the next steps:

  1. On the Amazon MSK console, select Replicators within the navigation pane.
  2. Select Create replicator.
  3. Within the Replicator particulars part, enter a reputation and elective description.

  1. Within the Supply cluster part, present the next data:
    1. For Cluster area, select us-east-1.
    2. For MSK cluster, enter the MSK cluster Amazon Useful resource Identify (ARN) for the Customary dealer.

After the supply cluster is chosen, it robotically selects the subnets related to the first cluster and the safety group related to the supply cluster. You may also choose further safety teams.

Make it possible for the safety teams have outbound guidelines to permit visitors to your cluster’s safety teams. Additionally be sure that your cluster’s safety teams have inbound guidelines that settle for visitors from the replicator safety teams offered right here.

  1. Within the Goal cluster part, for MSK cluster¸ enter the MSK cluster ARN for the Categorical dealer.

After the goal cluster is chosen, it robotically selects the subnets related to the first cluster and the safety group related to the supply cluster. You may also choose further safety teams.

Now let’s present the replicator settings.

  1. Within the Replicator settings part, present the next data:
    1. For the aim of the instance, we’ve got saved the subjects to copy as a default worth that may replicate all subjects from main to secondary cluster.
    2. For Replicator beginning place, we configure it to copy from the earliest offset, in order that we are able to get all of the occasions from the beginning of the supply subjects.
    3. To configure the subject identify within the secondary cluster as an identical to the first cluster, we choose Maintain the identical subject names for Copy settings. This makes positive that the MSK shoppers don’t want so as to add a prefix to the subject names.

    1. For this instance, we hold the Shopper Group Replication setting as default (be certain that it’s enabled to permit redirected shoppers resume processing information from the final processed offset).
    2. We set Goal Compression sort as None.

The Amazon MSK console will robotically create the required IAM insurance policies. Should you’re deploying utilizing the AWS Command Line Interface (AWS CLI), SDK, or AWS CloudFormation, you need to create the IAM coverage and use it as per your deployment course of.

  1. Select Create to create the replicator.

The method will take round 15–20 minutes to deploy the replicator. When the MSK replicator is operating, this will likely be mirrored within the standing.

Monitor replication

When the MSK replicator is up and operating, monitor the MessageLag metric. This metric signifies what number of messages are but to be replicated from the supply MSK cluster to the goal MSK cluster. The MessageLag metric ought to come right down to 0.

Migrate shoppers from supply to focus on cluster

When the MessageLag metric reaches 0, it signifies that each one messages have been replicated from the supply MSK cluster to the goal MSK cluster. At this stage, you’ll be able to minimize over consumer functions from the supply to the goal cluster. Earlier than initiating this step, affirm the well being of the goal cluster by reviewing the Amazon MSK metrics in Amazon CloudWatch and ensuring that the consumer functions are functioning correctly. Then full the next steps:

  1. Cease the producers writing information to the supply (outdated) cluster with Customary brokers and reconfigure them to put in writing to the goal (new) cluster with Categorical brokers.
  2. Earlier than migrating the shoppers, be sure that the MaxOffsetLag metric for the shoppers has dropped to 0, confirming that they’ve processed all current information within the supply cluster.
  3. When this situation is met, cease the shoppers and reconfigure them to learn from the goal cluster.

The offset lag occurs if the buyer is consuming slower than the speed the producer is producing information. The flat line within the following metric visualization reveals that the producer has stopped producing to the supply cluster whereas the buyer hooked up to it continues to eat the prevailing information and finally consumes all the information, due to this fact the metric goes to 0.

  1. Now you’ll be able to replace the bootstrap handle in properties and client.properties to level to the goal Categorical based mostly MSK cluster. You may get the bootstrap handle for IAM authentication from the MSK cluster (migration-express-broker-dest-cluster) on the Amazon MSK console beneath View Shopper Data.
export BS_TGT=<>
sed -i "s/BOOTSTRAP_SERVERS_CONFIG=.*/BOOTSTRAP_SERVERS_CONFIG=${BS_TGT}/g" producer.properties
sed -i "s/bootstrap.servers=.*/bootstrap.servers=${BS_TGT}/g" client.properties

  1. Run the clickstream producer to generate occasions within the clickstream subject:
cd /residence/ec2-user/clickstream-producer-for-apache-kafka/

java -jar goal/KafkaClickstreamClient-1.0-SNAPSHOT.jar -t clickstream 
-pfp /residence/ec2-user/producer.properties -nt 8 -rf 60 -iam 
-gsr -gsrr <> -grn default-registry -gar

  1. In one other Session Supervisor occasion and from that shell, run the clickstream client to eat from the subject:
cd /residence/ec2-user/clickstream-consumer-for-apache-kafka/

java -jar goal/KafkaClickstreamConsumer-1.0-SNAPSHOT.jar -t clickstream 
-pfp /residence/ec2-user/client.properties -nt 3 -rf 60 -iam 
-gsr -gsrr <> -grn default-registry

We will see that the producers and shoppers at the moment are producing and consuming to the goal Categorical based mostly MSK cluster. The producers and shoppers will run for 60 seconds earlier than they exit.

The next screenshot reveals producer-produced messages to the brand new Categorical based mostly MSK cluster for 60 seconds.

Migrate stateful functions

Stateful functions equivalent to Kafka Streams, KSQL, Apache Spark, and Apache Flink use their very own checkpointing mechanisms to retailer client offsets as an alternative of counting on Kafka’s client group offset mechanism. When migrating subjects from a supply cluster to a goal cluster, the Kafka offsets within the supply will differ from these within the goal. In consequence, migrating a stateful software together with its state requires cautious consideration, as a result of the prevailing offsets are incompatible with the goal cluster’s offsets. Earlier than migrating stateful functions, it’s essential to cease producers and be sure that client functions have processed all information from the supply MSK cluster.

Migrate Kafka Streams and KSQL functions

Kafka Streams and KSQL retailer client offsets in inside changelog subjects. It’s advisable to not replicate these inside changelog subjects to the goal MSK cluster. As a substitute, the Kafka Streams software ought to be configured to begin from the earliest offset of the supply subjects within the goal cluster. This permits the state to be rebuilt. Nonetheless, this technique leads to duplicate processing, as a result of all the information within the subject is reprocessed. Due to this fact, the goal vacation spot (equivalent to a database) should be idempotent to deal with these duplicates successfully.

Categorical brokers don’t permit configuring section.bytes to optimize efficiency. Due to this fact, the inner subjects have to be manually created earlier than the Kafka Streams software is migrated to the brand new Categorical based mostly cluster. For extra data, confer with Utilizing Kafka Streams with MSK Categorical brokers and MSK Serverless.

Migrate Spark functions

Spark shops offsets in its checkpoint location, which ought to be a file system suitable with HDFS, equivalent to Amazon Easy Storage Service (Amazon S3). After migrating the Spark software to the goal MSK cluster, you need to take away the checkpoint location, inflicting the Spark software to lose its state. To rebuild the state, configure the Spark software to begin processing from the earliest offset of the supply subjects within the goal cluster. This may result in re-processing all the information from the beginning of the subject and due to this fact will generate duplicate information. Consequently, the goal vacation spot (equivalent to a database) should be idempotent to successfully deal with these duplicates.

Migrate Flink functions

Flink shops client offsets throughout the state of its Kafka supply operator. When checkpoints are accomplished, the Kafka supply commits the present consuming offset to supply consistency between Flink’s checkpoint state and the offsets dedicated on Kafka brokers. In contrast to different programs, Flink functions don’t depend on the __consumer_offsets subject to trace offsets; as an alternative, they use the offsets saved in Flink’s state.

Throughout Flink software migration, one strategy is to begin the applying with no Savepoint. This strategy discards your complete state and reverts to studying from the final dedicated offset of the buyer group. Nonetheless, this prevents the applying from precisely rebuilding the state of downstream Flink operators, resulting in discrepancies in computation outcomes. To deal with this, you’ll be able to both keep away from replicating the buyer group of the Flink software or assign a brand new client group to the applying when restarting it within the goal cluster. Moreover, configure the applying to begin studying from the earliest offset of the supply subjects. This allows re-processing all information from the supply subjects and rebuilding the state. Nonetheless, this technique will end in duplicate information, so the goal system (equivalent to a database) should be idempotent to deal with these duplicates successfully.

Alternatively, you’ll be able to reset the state of the Kafka supply operator. Flink makes use of operator IDs (UIDs) to map the state to particular operators. When restarting the applying from a Savepoint, Flink matches the state to operators based mostly on their assigned IDs. It is strongly recommended to assign a singular ID to every operator to allow seamless state restoration from Savepoints. To reset the state of the Kafka supply operator, change its operator ID. Passing the operator ID as a parameter in a configuration file can simplify this course of. Restart the Flink software with parameter --allowNonRestoredState (if you’re operating self-managed Flink). This may reset solely the state of the Kafka supply operator, leaving different operator states unaffected. In consequence, the Kafka supply operator resumes from the final dedicated offset of the buyer group, avoiding full reprocessing and state rebuilding. Though this would possibly nonetheless produce some duplicates within the output, it leads to no information loss. This strategy is relevant solely when utilizing the DataStream API to construct Flink functions.

Conclusion

Migrating from a Customary dealer MSK cluster to an Categorical dealer MSK cluster utilizing MSK Replicator supplies a seamless, environment friendly transition with minimal downtime. By following the steps and methods mentioned on this publish, you’ll be able to reap the benefits of the high-performance, cost-effective advantages of Categorical brokers whereas sustaining information consistency and software uptime.

Able to optimize your Kafka infrastructure? Begin planning your migration to Amazon MSK Categorical brokers right now and expertise improved scalability, velocity, and reliability. For extra particulars, confer with the Amazon MSK Developer Information.

Concerning the Writer

Subham Rakshit is a Senior Streaming Options Architect for Analytics at AWS based mostly within the UK. He works with clients to design and construct streaming architectures to allow them to get worth from analyzing their streaming information. His two little daughters hold him occupied more often than not exterior work, and he loves fixing jigsaw puzzles with them. Join with him on LinkedIn.