Using S3 replication, you can set-up automatic replication of S3 objects from one bucket to another. The source and destination bucket can be within the same AWS account or in different accounts. You can also replicate objects from one source bucket to multiple destination buckets.
If you want to have a second copy of your objects in a different AWS region, you can create a replication rule to perform cross-region replication of S3 objects.
This tutorial has the following examples on how to setup and manage replication rules on S3 bucket using AWS s3api CLI:
- View Current Replication Rules on a S3 Bucket
- Delete All Replication Rules from a S3 Bucket
- Add New Replication Rule on a S3 Bucket with Default Values
- Replication Rule with Custom Rule Name
- Replication Rule with a specific S3 Object Prefix Value
- Replication Rule based on S3 Object Tag Value
- Replication Rule based on both S3 Object Prefix AND Object Tag Values
- Disable an Existing Replication Rule on a S3 Bucket
- Replication Rule to Replicate S3 KMS Encrypted Objects
- Replication Rule with a Different Storage Class on Destination
- Replication Rule for Cross Account (and Cross Region) S3 Buckets
- Replication Rule with RTC Enabled
- Replication Rule Combined with everything from above (Cross Region, Cross Account, Encryption, Tags/Prefix, RTC)
1. View Current Replication Rules on an S3 Bucket
If you a new to managing S3 from AWS CLI, refer to this: Essential AWS S3 CLI Command Examples to Manage Buckets and Objects
For all replication related activities, we’ll be using the s3api command in the AWS CLI.
Before we get started, to view all existing replications rules on an S3 bucket, use s3api get-bucket-replication option as shown below.
As the following output shows, this bucket already has a few replication rules set up on it.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_backup-dest",
"Rules": [
{
"ID": "mytechmint-replication-with-kms",
"Priority": 3,
"Filter": {},
"Status": "Enabled",
"SourceSelectionCriteria": {
"SseKmsEncryptedObjects": {
"Status": "Enabled"
}
},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest",
"EncryptionConfiguration": {
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:111111111111:key/22222222-dddd-4444-aaaa-555555555555"
}
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
},
{
"ID": "mytechmint-backup-replication",
"Priority": 2,
"Filter": {},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
},
{
"ID": "mytechmint-replication",
"Priority": 1,
"Filter": {},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
If you don’t have any replication rules on your bucket, you’ll get the following message.
$ aws s3api get-bucket-replication --bucket mytechmint-source An error occurred (ReplicationConfigurationNotFoundError) when calling the GetBucketReplication operation: The replication configuration was not found
If you don’t specify a bucket name in the CLI, you’ll get this: aws: error: the following arguments are required: –bucket
2. Delete All Replication Rules from a S3 Bucket
For this example, let us delete all the existing replication rules and start clean. Use s3api delete-bucket-replication option to delete all existing replication rules from a S3 bucket.
aws s3api delete-bucket-replication --bucket mytechmint-source
Verify that all the existing replication rules are deleted as shown below.
aws s3api get-bucket-replication --bucket mytechmint-source
An error occurred (ReplicationConfigurationNotFoundError) when calling the GetBucketReplication operation: The replication configuration was not found
3. Add New Replication Rule on a S3 Bucket with Default Values
To add a new replication rule, first create a replication json file that contains the details about the replication as shown below.
$ cat /project/rep3.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": ""},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
}
}
]
}
In the above, the Role contains the ARN of the IAM role that S3 can assume to replicate objects on your behalf. If you’ve setup a replication rule from the console before, you should already have this role created for you, and you can reuse that role here.
In the Rules section, make sure to specify the ARN for the destination S3 bucket.
Once the replication JSON file is ready, use the s3api put-bucket-replication option as shown below to create the replication rule on your source S3 bucket.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep3.json
Verify that the replication rule is created successfully as shown below.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "YYYYYY22222MMMMMMMMzzzzzllllllccccccttttttZZZZZx",
"Priority": 1,
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
4. Replication Rule with Custom Rule Name
In the above output, the ID field is what will be displayed as “Rule Name” in the S3 console. If you don’t specify a rule name in the json file, you’ll get a random very long ID assigned as “Rule Name”.
To specify your own custom rule name while creating a replication rule, use the “ID” field in the json file as shown below to use your custom name.
$ cat rep4.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": ""},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
}
}
]
}
Create the rule with the custom rule name using s3api put-bucket-replication option.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep4.json
As you see from the following output, the “ID” has the custom rule name now.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Priority": 1,
"Filter": {
"Prefix": ""
},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
5. Replication Rule with a specific S3 Object Prefix Value
Sometimes you might not want to replicate all the objects from source S3 bucket to destination bucket.
In that case, you can selectively replicate objects using their prefix values.
For this, in the JSON file, use the “Filter” field to specify the Prefix as shown below.
$ cat rep5.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": "project/data1/"},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
}
}
]
}
The following example will replicate only the S3 objects matching project/data1/ prefix from source to destination.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep5.json
Once the replication rule is created, verify that the rule has the prefix filter that you specified in the JSON file as shown below.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Priority": 1,
"Filter": {
"Prefix": "project/data1/"
},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
6. Replication Rule based on S3 Object Tag Value
Sometimes you may want to replicate objects not based on prefix, but based on the Tags attached to it.
For this, in the “Filter” field in JSON objects, specify one or more Tags as shown below.
$ cat rep6.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Tag": { "Key": "Name", "Value": "Development" } },
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
}
}
]
}
The following will create a replication rule to replicate only the S3 objects that has the value for the Tag “Name” as “Development”.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep6.json
Once the replication rule is created, verify that the rule has the filter with the Tags that you specified as shown below.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Priority": 1,
"Filter": {
"Tag": {
"Key": "Name",
"Value": "Development"
}
},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
7. Replication Rule based on both S3 Object Prefix AND Object Tag Values
You can also combine both Prefix and Tag filters by using “And” inside the “Filter” field in your JSON file.
$ cat rep7.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : {
"And": {
"Prefix": "data/production",
"Tags": [
{
"Key": "Name",
"Value": "Development"
}
]
}
},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
}
}
]
}
The following will create a replication rule to replicate only the S3 objects that has both the prefix “data/production” and the value for the Tag “Name” is “Development”.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep7.json
Once the replication rule is created, verify that the rule has the combination filter with both the Prefix the Tags that you specified as shown below.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Priority": 1,
"Filter": {
"And": {
"Prefix": "data/production",
"Tags": [
{
"Key": "Name",
"Value": "Development"
}
]
}
},
"Status": "Enabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
8. Disable an Existing Replication Rule on a S3 Bucket
Instead of deleting a replication rule, you can also temporarily disable it by setting the “Status” field as “Disabled” in your JSON file as shown below.
$ cat rep8.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Disabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": ""},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
}
}
]
}
The following command will disable the existing replication rule.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep8.json
Verify that the rule is disabled as shown below.
$ aws s3api get-bucket-replication --bucket mytechmint-source
{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Priority": 1,
"Filter": {
"Prefix": ""
},
"Status": "Disabled",
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest"
},
"DeleteMarkerReplication": {
"Status": "Disabled"
}
}
]
}
}
9. Replication Rule to Replicate S3 KMS Encrypted Objects
When you are replicating S3 objects that are encrypted with KMS, you should specify the “SseKmsEncryptedObjects” for the source with status as enabled, and for the destination specify the ReplicaKmsKeyID as shown in the following JSON file.
$ cat rep9.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": "project/data1/"},
"SourceSelectionCriteria": {
"SseKmsEncryptedObjects": {
"Status": "Enabled"
}
},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest",
"EncryptionConfiguration": {
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:111111111111:key/22222222-dddd-4444-aaaa-555555555555"
}
}
}
]
}
The following will replicate the S3 objects that are encrypted using KMS keys from source to destination bucket as specified in the JSON file above.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep9.json
When you specify the SseKmsEncryptedObjects for source, but don’t specify the ReplicaKmsKeyID for the destination bucket, you’ll get this error: An error occurred (InvalidRequest) when calling the PutBucketReplication operation: ReplicaKmsKeyID must be specified if SseKmsEncryptedObjects tag is present.
If the KMS key you specify doesn’t exist, you’ll get this error: An error occurred (InvalidArgument) when calling the PutBucketReplication operation: Invalid ReplicaKmsKeyID ARN.
10. Replication Rule with a Different Storage Class on Destination
While replicating objects from one bucket to another, you can also change the storage class on the destination S3 bucket. This is very helpful when you are replicating object to a bucket in a different region for DR purpose.
In the following example, the source bucket is on standard S3 storage class, but while replicating to the destination bucket, we are storing the object at the destination in S3 Standard-Infrequent Access storage class.
$ cat rep10.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": "project/data1/"},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest",
"StorageClass": "STANDARD_IA"
}
}
]
}
The following will create the replication rule based on the above JSON file to set a different storage class at the destination bucket.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep10.json
11. Replication Rule for Cross Account (and Cross Region) S3 Buckets
If you want to replicate S3 objects to a destination bucket that is owned by another account, specify the target AWS account number under the “Destination” section. In this example, the destination bucket can also be on a different region than the source bucket to achieve your cross account and cross region replication requirement.
$ cat rep11.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": "project/data1/"},
"Destination": {
"Bucket": "arn:aws:s3:::backup-dest",
"Account": "222222222222"
}
}
]
}
Execute the following command to create a replication rule which will perform cross-account replication to the account number specified in the above JSON file.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep11.json
12. Replication Rule with RTC Enabled
RTC stands for Replication Time Control. S3 RTC allows you to complete the replication of 99.99 percent of objects within 15 minutes. There is an additional cost to it, please refer to the S3 pricing for more details.
To enable RTC in your replication rule, add both Metrics and ReplicationTime block to your Destination section in the JSON file as shown below. Currently, the only value that you can provide is 15 Minutes in this JSON file.
$ cat rep12.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : { "Prefix": "project/data1/"},
"Destination": {
"Bucket": "arn:aws:s3:::mytechmint-dest",
"ReplicationTime": {
"Status": "Enabled",
"Time": {
"Minutes": 15
}
},
"Metrics": {
"Status": "Enabled",
"EventThreshold": {
"Minutes": 15
}
}
}
}
]
}
The following will create the replication rule with the RTC enable based on the above JSON file.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep12.json
If you don’t specify the Time, in the JSON file, you’ll get this error: Parameter validation failed: Missing required parameter in ReplicationConfiguration.Rules[0].Destination.ReplicationTime: “Time”
If you specify a Time value of anything other than 15 Minutes, you’ll get this error: An error occurred (InvalidArgument) when calling the PutBucketReplication operation: Invalid time minute value
If you have only ReplicationTime but not the Metrics in your JSON file, you’ll get this error: An error occurred (InvalidRequest) when calling the PutBucketReplication operation: Replication destination must contain both ReplicationTime and Metrics or neither.
13. Replication Rule with multiple options (Cross Region, Cross Account, Encryption, Tags/Prefix, RTC)
You can combine one of more options from the previous examples in one replication rule. The following example has prefix and tag filters, cross-region with source and destination bucket in different regions, cross account with destination bucket in a different account than source bucket, replicate objects that are encrypted with KMS key, and enable replication time control.
$ cat rep13.json
{
"Role": "arn:aws:iam::111111111111:role/service-role/s3crr_role_for_mytechmint-source_to_mytechmint-dest",
"Rules": [
{
"ID": "mytechmint-dev-to-prod-rule1",
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"Filter" : {
"And": {
"Prefix": "data/production",
"Tags": [
{
"Key": "Name",
"Value": "Development"
}
]
}
},
"SourceSelectionCriteria": {
"SseKmsEncryptedObjects": {
"Status": "Enabled"
}
},
"Destination": {
"Bucket": "arn:aws:s3:::backup-dest",
"Account": "222222222222",
"EncryptionConfiguration": {
"ReplicaKmsKeyID": "arn:aws:kms:us-west-2:111111111111:key/22222222-dddd-4444-aaaa-555555555555"
},
"ReplicationTime": {
"Status": "Enabled",
"Time": {
"Minutes": 15
}
},
"Metrics": {
"Status": "Enabled",
"EventThreshold": {
"Minutes": 15
}
}
}
}
]
}
The following will create the replication rule based on the above JSON file that contains multiple replication options set in the rule.
aws s3api put-bucket-replication --bucket mytechmint-source \ --replication-configuration file:///project/rep13.json
