Migrate third-party and self-managed Apache Kafka clusters to Amazon MSK Categorical brokers with Amazon MSK Replicator

Migrating Apache Kafka workloads to the cloud typically entails managing advanced replication infrastructure, coordinating utility cutovers with prolonged downtime home windows, and sustaining deep experience in open-source instruments like Apache Kafka’s MirrorMaker 2 (MM2). These challenges decelerate migrations and improve operational danger. Amazon MSK Replicator addresses these challenges, enabling you emigrate your Kafka deployments (known as “exterior” Kafka clusters) to Amazon MSK Categorical brokers with minimal operational overhead and lowered downtime. MSK Replicator helps information migration from Kafka deployments (model 2.8.1 or later) which have SASL/SCRAM authentication enabled – together with Kafka clusters operating on-premises, on AWS, or different cloud suppliers, in addition to Kafka-protocol-compatible companies like Confluent Platform, Avien, RedPanda, WarpStream, or AutoMQ when configured with SASL/SCRAM authentication.

On this submit, we stroll you thru learn how to replicate Apache Kafka information out of your exterior Apache Kafka deployments to Amazon MSK Categorical brokers utilizing MSK Replicator. You’ll learn to configure authentication in your exterior cluster, set up community connectivity, arrange bidirectional replication, and monitor replication well being to attain a low-downtime migration.

The way it works

MSK Replicator is a totally managed serverless service that replicates matters, configurations, and offsets from cluster to cluster. It alleviates the necessity to handle advanced infrastructure or configure open-source instruments.

Earlier than MSK Replicator, clients used instruments like MM2 for migrations. These instruments lack bi-directional matter replication when utilizing the identical matter names, creating advanced utility architectures to devour completely different matters on completely different clusters. Customized replication insurance policies in MM2 can enable equivalent matter names, however MM2 nonetheless lacks bidirectional offset replication as a result of the MM2 structure requires producers and shoppers to run on the identical cluster to copy offsets. This created advanced migrations that required both migrating shoppers earlier than producers or big-bang migrations migrating all purposes directly. When clients run into points in the course of the migration, the rollback course of is error-prone and introduces giant quantities of duplicate message processing because of the lack of client group offset synchronization. These approaches create danger and complexity for purchasers that make migrations tough to handle.

MSK Replicator addresses these issues by supporting bidirectional replication of information and enhanced client group offset synchronization. MSK Replicator copies matters and offsets from an exterior Kafka cluster to MSK, permitting you to protect the identical matter and client group names on each clusters. MSK Replicator additionally helps making a second Replicator occasion for bidirectional replication of each information and enhanced offset synchronization, permitting producers and shoppers to run independently on completely different Kafka clusters. Information printed or consumed on the Amazon MSK cluster might be replicated again to the exterior cluster by the second Replicator. This characteristic works when producers and shoppers are migrated no matter order with out worrying about dependencies between purposes.

As a result of MSK Replicator offers bidirectional information replication and enhanced client group offset synchronization, you’ll be able to transfer producers and shoppers at your personal tempo with out information loss. This reduces migration complexity, permitting you emigrate purposes between your exterior Kafka cluster and Amazon MSK no matter order. Should you run into issues in the course of the migration, enhanced offset synchronization permits you to roll again modifications by shifting purposes again to the exterior Kafka cluster, the place they restart from the newest checkpoint from the Amazon MSK cluster.

For instance, contemplate three purposes:

1. The “Orders” utility, which accepts incoming orders and writes them to the orders Kafka matter
2. The “Order standing” utility, which reads from the “orders” Kafka matter and writes standing updates to the order_status matter
3. The “Buyer notification” utility, which reads from the order_status matter and notifies clients when standing modifications

MSK Replicator allows these purposes to be migrated between an on-premises Apache Kafka cluster and an Amazon MSK Categorical cluster with low downtime and no information loss, no matter order. The “Order standing” utility can migrate first, obtain orders from the on-premises “Orders” utility, and ship standing updates to the on-premises “Buyer notification” utility. If points come up in the course of the migration, the “Order standing” utility can roll again to the on-premises cluster and its client group offsets for the orders matter might be prepared for it to choose up from the place it left off on the Amazon MSK cluster.

MSK Replicator helps information distribution throughout hybrid and multi-cloud environments for analytics, compliance, and enterprise continuity. It is usually configured for catastrophe restoration eventualities the place Amazon MSK Categorical serves as a resilient goal in your exterior Kafka clusters.

In case you are at present utilizing MM2 for replication, see Amazon MSK Replicator and MirrorMaker2: Choosing the proper replication technique for Apache Kafka catastrophe restoration and migrations to know which resolution most closely fits your use case.

Resolution overview

MSK Replicator helps Kafka deployments operating model 2.8.1 or later as a supply, together with third social gathering managed Kafka companies, self-managed Kafka, and on-premises or third-party cloud-hosted Kafka. MSK Replicator routinely handles information switch, makes use of SASL/SCRAM authentication with SSL encryption, and maintains client group positions throughout each clusters. If you don’t use SASL/SCRAM right now, this may be configured as a brand new listener used for MSK Replicator permitting present shoppers to make use of their present authentication mechanisms alongside MSK Replicator.

Conditions

To comply with together with this walkthrough, you want the next assets in place:

Establishing replication

Step 1: Configure community connectivity

You’ll be able to arrange community connectivity between your exterior Kafka cluster and your AWS VPC utilizing strategies reminiscent of AWS Direct Join for devoted community connections, AWS Website-to-Website VPN for encrypted connections over the web, and AWS VPC peering or AWS Transit Gateway for connections between AWS VPCs. Confirm that IP routing and DNS decision are correctly configured between your exterior cluster and AWS.

To confirm IP routing and DNS decision, connect with your exterior Kafka cluster from inside your VPC through the use of the Kafka CLI to record matters on the exterior cluster. Should you can record matters out of your VPC utilizing the Kafka CLI, this implies DNS decision and IP routing are working efficiently. If it fails, work together with your community admins to troubleshoot community connectivity points.

Step 2: Configure exterior cluster

On this step, you’ll arrange authentication in your exterior Kafka cluster and retailer the credentials in AWS Secrets and techniques Supervisor in order that MSK Replicator can join securely.

Configure authentication

Utilizing the exterior cluster admin consumer, configure SASL/SCRAM authentication for MSK Replicator utilizing SHA-256 or 512 in your exterior Kafka cluster. Create a SASL/SCRAM consumer for MSK Replicator and provides the consumer the next ACL permissions:

• Subject operations – Alter, AlterConfigs, Create, Describe, DescribeConfigs, Learn, Write
• Group operations – Learn, Describe
• Cluster operations – Create, ClusterAction, Describe, DescribeConfigs

Configure SecretsManager

AWS Secrets and techniques Supervisor shops your SASL/SCRAM credentials securely in order that MSK Replicator can retrieve them at runtime. The key should use JSON format and have the next keys:

• username – The SCRAM username that you just configured within the authentication step above
• password – The SCRAM password that you just configured within the authentication step above
• certificates – The general public root CA certificates (the top-level certificates authority that issued your cluster’s TLS certificates) and the intermediate CA chain (intermediate certificates between the basis and your cluster’s certificates), used for SSL handshakes with the exterior cluster

Optionally, you might create separate secrets and techniques for SCRAM credentials and the SSL certificates. This strategy is helpful when secrets and techniques for SCRAM credentials and certificates are provisioned in numerous phases, reminiscent of in Infrastructure as Code (IaC) pipelines.

Retrieve the cluster ID

Because the admin consumer, use the Kafka CLI instruments to retrieve the cluster ID of your exterior cluster. Run the next command, changing your-broker-host:9096 with the deal with of one among your exterior cluster’s bootstrap servers:

bin/kafka-cluster.sh cluster-id --bootstrap-server your-broker-host:9096 --config admin.properties

The command returns a cluster ID string reminiscent of lkc-abc123. Be aware of this worth as a result of you will have it when creating the replicator in Step 4.

Step 3: Create your MSK Categorical goal cluster

Together with your exterior cluster configured, now you can arrange the goal. Create an Amazon MSK Categorical cluster with IAM authentication enabled. Guarantee that the cluster is in subnets which have entry to AWS Secrets and techniques Supervisor endpoints. See Get began utilizing Amazon MSK for extra info on creating an MSK cluster.

Step 4: Create the replicator

Now that each clusters are prepared, you’ll be able to join them by establishing the MSK Replicator with the suitable IAM position and replication configuration.

Arrange an IAM position for MSK Replicator

MSK Replicator wants an IAM position to work together together with your MSK Categorical cluster and retrieve secrets and techniques. Arrange a service execution IAM position with a belief coverage permitting kafka.amazonaws.com and fasten the AWSMSKReplicatorExecutionRole permissions coverage. Be aware of the position ARN for creating the replicator.

Create and fasten a coverage for accessing your Secrets and techniques Supervisor secrets and techniques and studying/writing information in your MSK cluster. See Creating roles and attaching insurance policies (console) for extra info on creating IAM roles and insurance policies.

The next is an instance coverage for studying and writing information to your MSK cluster and studying KMS-encrypted Secrets and techniques Supervisor secrets and techniques:

{ 
    "Model": "2012-10-17", 
    "Assertion": [ 
        { 
            "Sid": "SecretsManagerAccess", 
            "Effect": "Allow", 
            "Action": [ 
                "secretsmanager:GetSecretValue", 
                "secretsmanager:DescribeSecret" 
            ], 
            "Useful resource": [ 
                "", 
                "" 
            ] 
        }, 
        { 
            "Sid": "KMSDecrypt", 
            "Impact": "Permit", 
            "Motion": "kms:Decrypt", 
            "Useful resource": "" 
        }, 
        { 
            "Sid": "TargetClusterAccess", 
            "Impact": "Permit", 
            "Motion": [ 
                "kafka-cluster:Connect", 
                "kafka-cluster:DescribeCluster", 
                "kafka-cluster:AlterCluster", 
                "kafka-cluster:DescribeClusterDynamicConfiguration", 
                "kafka-cluster:AlterClusterDynamicConfiguration", 
                "kafka-cluster:DescribeTopic", 
                "kafka-cluster:CreateTopic", 
                "kafka-cluster:AlterTopic", 
                "kafka-cluster:DescribeTopicDynamicConfiguration", 
                "kafka-cluster:AlterTopicDynamicConfiguration", 
                "kafka-cluster:WriteData", 
                "kafka-cluster:WriteDataIdempotently", 
                "kafka-cluster:ReadData", 
                "kafka-cluster:DescribeGroup", 
                "kafka-cluster:AlterGroup" 
            ], 
            "Useful resource": [ 
                "arn:aws:kafka:::cluster/*/*", 
                "arn:aws:kafka:::topic//*", 
                "arn:aws:kafka:::group/*/*" 
            ] 
        }, 
        { 
            "Sid": "CloudWatchLogsAccess", 
            "Impact": "Permit", 
            "Motion": [ 
                "logs:CreateLogStream", 
                "logs:PutLogEvents", 
                "logs:DescribeLogStreams" 
            ], 
            "Useful resource": "" 
        } 
    ] 
}

Create the replicator for exterior to MSK replication

Use the AWS CLI, API, or Console to create your replicator. Right here’s an instance utilizing the AWS CLI:

aws kafka create-replicator 
  --replicator-name external-to-msk 
  --service-execution-role-arn "arn:aws:iam::123456789012:position/MSKReplicatorRole" 
  --kafka-clusters file://./kafka-clusters.json 
  --replication-info-list file://./replication-info.json 
  --log-delivery file://./log-delivery.json 
  --region us-east-1

The kafka-clusters.json file defines the supply and goal Kafka cluster connection info, replication-info.json specifies which matters to copy and learn how to deal with client group offset synchronization, and log-delivery.json specifies the CloudWatch logging configuration. The next tables describe the required parameters:

CLI inputs:

CLI Parameter	Description	Instance
replicator-name	The title of the replicator	external-to-msk
service-execution-role-arn	The ARN for the service execution IAM position you created	arn:aws:iam::123456789012:position/MSKReplicatorRole
kafka-clusters	The Kafka cluster connection data	See under
replication-info-list	The replication configuration	See under
log-delivery	The logging configuration	See under

Key kafka-clusters.json inputs:

CLI Parameter	Description	Instance
ApacheKafkaClusterId	The cluster ID retrieved in Step 2	lkc-abc123
RootCaCertificate	The Secrets and techniques Supervisor ARN containing the general public CA certificates and intermediate CA chain	arn:aws:secretsmanager:::secret:my-cert
MskClusterArn	The ARN for the MSK Categorical cluster	arn:aws:kafka:::cluster/my-cluster/abc-123
SecretArn	The Secrets and techniques Supervisor ARN containing the SASL/SCRAM username and password	arn:aws:secretsmanager:::secret:my-creds
SecurityGroupIds	The safety group IDs for MSK Replicator	sg-0123456789abcdef0

Key replication-info.json inputs:

CLI Parameter	Description	Instance
TargetCompressionType	The compression sort to make use of for replicating information	LZ4
TopicsToReplicate	The record of matters to copy (use [“.*”] for all matters)	[“my-topic”]
ConsumerGroupsToReplicate	The record of client teams to copy	[“my-group”]
StartingPosition	The purpose within the Kafka matters to start replication from (both EARLIEST or LATEST)	EARLIEST
ConsumerGroupOffsetSyncMode	Whether or not or to not use enhanced bidirectional client group offset synchronization	ENHANCED

Notice that startingPosition is about to EARLIEST within the configuration under, which implies the replicator begins studying from the oldest obtainable offset on every matter. That is the beneficial setting for migrations to keep away from information loss.

Key log-delivery.json inputs:

CLI Parameter	Description	Instance
Enabled	Means that you can allow CloudWatch logging	true
LogGroup	The CloudWatch logs log group title to log to	/msk/replicator/my-replicator

Extra log supply strategies for Amazon S3 and Amazon Information Firehose are supported. On this submit, we use CloudWatch logging.

The configs ought to appear like the next for exterior to MSK replication.

kafka-clusters.json:

[ 
  { 
    "ApacheKafkaCluster": { 
      "ApacheKafkaClusterId": "lkc-abc123", 
      "BootstrapBrokerString": "broker1.example.com:9096" 
    }, 
    "ClientAuthentication": { 
      "SaslScram": { 
        "Mechanism": "SHA512", 
        "SecretArn": "arn:aws:secretsmanager:::secret:my-creds" 
      } 
    }, 
    "EncryptionInTransit": { 
      "EncryptionType": "TLS", 
      "RootCaCertificate": "arn:aws:secretsmanager:::secret:my-cert" 
    } 
  }, 
  { 
    "AmazonMskCluster": { 
      "MskClusterArn": "arn:aws:kafka:::cluster/my-cluster/abc-123" 
    }, 
    "VpcConfig": { 
      "SecurityGroupIds": ["sg-0123456789abcdef0"], 
      "SubnetIds": ["subnet-abc123", "subnet-abc124", "subnet-abc125"] 
    } 
  } 
] 

replication-info.json: 

[ 
  { 
    "SourceKafkaClusterId": "lkc-abc123", 
    "TargetKafkaClusterArn": "arn:aws:kafka:::cluster/my-cluster/abc-123", 
    "TargetCompressionType": "LZ4", 
    "TopicReplication": { 
      "TopicsToReplicate": ["my-topic"], 
      "CopyTopicConfigurations": true, 
      "CopyAccessControlListsForTopics": true, 
      "DetectAndCopyNewTopics": true, 
      "StartingPosition": {"Kind": "EARLIEST"}, 
      "TopicNameConfiguration": {"Kind": "IDENTICAL"} 
    }, 
    "ConsumerGroupReplication": { 
      "ConsumerGroupsToReplicate": ["my-group"], 
      "SynchroniseConsumerGroupOffsets": true, 
      "DetectAndCopyNewConsumerGroups": true, 
      "ConsumerGroupOffsetSyncMode": "ENHANCED" 
    } 
  } 
] 

log-delivery.json: 

{ 
  "ReplicatorLogDelivery": {
     "CloudWatchLogs": {
       "Enabled": true, 
       "LogGroup": ""
     }
  } 
}

Configure bidirectional replication from MSK to the exterior cluster

To allow bidirectional replication, create a second replicator that replicates in the other way. Use the identical IAM position and community configuration from Step 4, however swap the supply and goal. Substitute SourceKafkaClusterId with TargetKafkaClusterId and TargetKafkaClusterArn with SourceKafkaClusterArn in a brand new msk-to-external-replication-info.json file:

aws kafka create-replicator 
  --replicator-name msk-to-external 
  --service-execution-role-arn "arn:aws:iam::123456789012:position/MSKReplicatorRole" 
  --kafka-clusters file:///./kafka-clusters.json 
  --replication-info-list file:///./msk-to-external-replication-info.json 
  --log-delivery file:///./log-delivery.json 
  --region us-east-1

Monitoring replication well being

Monitor your replication utilizing Amazon CloudWatch metrics. Three key metrics to know are MessageLag, SumOffsetLag, and ReplicationLatency. MessageLag measures how far behind the replicator is from the exterior cluster when it comes to messages not but replicated, whereas SumOffsetLag measures how far behind a client group is from the newest message in a subject. ReplicationLatency is the quantity of latency between the supply and goal clusters in information replication. When the three attain a sustained low degree, your clusters are absolutely synchronized for each information and client group offsets.

To troubleshoot MSK Replicator replication or errors, use the CloudWatch logs to get extra particulars concerning the well being of the replicator. MSK Replicator logs standing and troubleshooting info which could be useful in diagnosing points like connectivity, authentication, and SSL errors.

Notice that the replication is asynchronous, so there might be some lag throughout replication. The lag will attain zero as soon as a shopper is shut down throughout migration to the goal cluster. This takes about 30 seconds beneath regular operations, permitting a low downtime migration with out information loss. In case your lag is frequently growing or doesn’t attain a sustained low degree, this means that you’ve inadequate partitions for high-throughput replication. Check with Troubleshoot MSK Replicator for extra info on troubleshooting replication throughput and lag.

Key metrics embrace:

• MessageLag – Screens the sync between the MSK Replicator and the supply cluster. MessageLag signifies the lag between the messages produced to the supply cluster and messages consumed by the replicator. It isn’t the lag between the supply and goal cluster.
• ReplicationLatency – Time taken for data to copy from supply to focus on cluster (ms)
• ReplicatorThroughput – Common variety of bytes replicated per second
• ReplicatorFailure – Variety of failures the replicator is experiencing
• KafkaClusterPingSuccessCount – Connection well being indicator (1 = wholesome, 0 = unhealthy)
• ConsumerGroupCount – Whole client teams being synchronized
• ConsumerGroupOffsetSyncFailure – Failures throughout offset synchronization
• AuthError – Variety of connections with failed authentication per second, by cluster
• ThrottleTime – Common time in ms a request was throttled by brokers, by cluster
• SumOffsetLag – Aggregated offset lag throughout partitions for a client group on a subject (MSK cluster-level metric)

For extra particulars on these metrics, see the MSK Replicator metrics documentation.

Your purposes are able to migrate when the next situations are met. For many workloads, you need to count on these metrics to stabilize inside just a few hours of beginning replication. Excessive-throughput clusters might take longer relying on matter quantity and partition depend.

• ReplicatorFailure = 0
• ConsumerGroupOffsetSyncFailure = 0
• KafkaClusterPingSuccessCount = 1 for each supply and goal clusters
• MessageLag
• Your sustained lag could also be decrease or greater relying in your throughput per partition, message dimension, and different components
• Sustained excessive message lag often signifies inadequate partitions for high-throughput replication

ReplicationLatency

Your sustained latency could also be decrease or greater relying in your throughput per partition, message dimension, and different components

Sustained excessive latency often signifies inadequate partitions for high-throughput replication

SumOffsetLag is at a sustained low degree on each clusters

• Offset values on the 2 clusters is probably not numerically equivalent.
• MSK Replicator interprets offsets between clusters so that customers resume from the proper place, however the uncooked offset numbers can differ on account of how offset translation works. What issues is that SumOffsetLag is at a sustained low degree.

ConsumerGroupCount (MSK) = Anticipated depend (exterior cluster)

• If ConsumerGroupCount is zero or doesn’t match the anticipated depend, then there is a matter within the Replicator configuration or a permissions difficulty stopping client group synchronization

Migrating your purposes

With bidirectional client offset synchronization, you’ll be able to migrate your producers and shoppers no matter order. Begin by monitoring replication metrics till they attain the goal values described within the earlier part. Then migrate your purposes (producers or shoppers) to make use of the MSK Categorical cluster endpoints and confirm that they’re producing and consuming as anticipated. Should you encounter points, you’ll be able to roll again by switching purposes again to the exterior cluster. The buyer offset synchronization makes positive that your purposes resume from their final dedicated place no matter which cluster they connect with.

For a complete, hands-on walkthrough of the end-to-end migration course of, discover the MSK Migration Workshop, which offers step-by-step steering for migrating your Kafka workloads to Amazon MSK.

Safety concerns

MSK Replicator makes use of SASL/SCRAM authentication with SSL encryption for safe information switch between your exterior cluster and AWS. The answer helps each publicly trusted certificates and personal or self-signed certificates. Credentials are saved securely in AWS Secrets and techniques Supervisor, and the goal MSK Categorical cluster makes use of IAM authentication for entry management.

When configuring safety, preserve the next in thoughts:

• Guarantee that the IAM position you create in Step 4 follows the precept of least privileges. Solely connect AWSMSKReplicatorExecutionRole and an IAM coverage for Secrets and techniques Supervisor with least-privileges entry to learn secret values and keep away from including broader permissions.
• Confirm that your Secrets and techniques Supervisor secret is encrypted with an AWS KMS key that the MSK Replicator service execution position has permission to decrypt.
• Affirm that the safety teams assigned to MSK Replicator enable outbound visitors to your exterior cluster’s dealer ports (usually 9096 for SASL/SCRAM with TLS) and to the MSK Categorical cluster.
• Rotate your SASL/SCRAM credentials periodically and replace the corresponding Secrets and techniques Supervisor secret. MSK Replicator picks up the brand new credentials routinely on the subsequent connection try.

Below the AWS shared accountability mannequin, AWS is liable for securing the underlying infrastructure that runs MSK Replicator, together with the compute, storage, and networking assets. You’re liable for configuring authentication mechanisms (SASL/SCRAM), managing credentials in AWS Secrets and techniques Supervisor, configuring community safety (safety teams and VPC settings), implementing IAM insurance policies following least privilege, and rotating credentials. For extra info, see Safety in Amazon MSK within the Amazon MSK Developer Information.

Cleanup

To keep away from ongoing fees, delete the assets you created throughout this walkthrough. Begin by deleting the replicators first, as a result of they depend upon the opposite assets:

aws kafka delete-replicator --replicator-arn

After each replicators are deleted, you’ll be able to take away the next assets in the event that they have been created solely for this walkthrough:

1. The MSK Categorical cluster (deleting a cluster additionally removes its saved information, so confirm that your purposes have absolutely migrated earlier than continuing)
2. The Secrets and techniques Supervisor secrets and techniques containing your SASL/SCRAM credentials and certificates
3. The IAM position and insurance policies created for MSK Replicator

You’ll be able to confirm {that a} replicator has been absolutely deleted by operating aws kafka list-replicators and confirming it now not seems within the output.

Conclusion

Amazon MSK Replicator simplifies the method of migrating to Amazon MSK Categorical brokers and establishes hybrid Kafka architectures. The absolutely managed service alleviates the operational complexity of managing replication whereas bidirectional client offset synchronization allows versatile, low-risk utility migration.

Subsequent Steps

To get began utilizing MSK Replicator emigrate purposes to MSK Categorical brokers, use the MSK Migration Workshop for a hands-on, end-to-end migration walkthrough. The Amazon MSK Replicator documentation contains detailed configuration particulars to assist configure MSK Replicator in your use case. From there, use MSK Replicator emigrate your Apache Kafka workloads to MSK Categorical dealer.

As soon as your migration is full, contemplate exploring multi-region replication patterns for catastrophe restoration, or integrating your MSK Categorical cluster with AWS analytics companies reminiscent of Amazon Information Firehose and Amazon Athena. Should you need assistance planning your migration, attain out to your AWS account group, AWS Assist or AWS Skilled Companies.

---

In regards to the authors