How to enable replication of delete marker from AWS S3 source bucket to destination bucket

Description

Deletion of items from AWS S3 source bucket mentioned on Amazon Simple Storage Service (S3) partition does not replicate the deletion of items to the destination bucket if replication is enabled on the source bucket.

After replication is enabled on the source bucket, by default, the S3 replication behavior V2 version is used, which does not replicate delete markers from the source bucket to the destination bucket. This behavior of AWS S3 replication has an impact on Enterprise Vault on user deletion and storage expiry. 

This article describes how to configure replication on the AWS S3 bucket and enable replication of delete marker from the source bucket to the destination bucket.

Prerequisites

Ensure that the following prerequisites are met:

• Both the source bucket and the destination bucket must have AWS S3 versioning enabled.
• As buckets can be created in the same region as well as different regions, both the source and destination buckets must have permissions for replication between them.
• AWS CLI or AWS PowerShell is installed on the local computer to run commands mentioned in this article.

Solution

Perform the following steps:

1. Ensure that you have created the AWS S3 source and destination buckets as per your business requirements.
2. Enable versioning on both AWS S3 source and destination buckets.
3. Configure the replication rules on the source bucket using a new or existing IAM role.
4. Irrespective of whether a new IAM role is created or an existing IAM role is used, ensure that the following actions and permissions are mentioned in the policy attached to the IAM role. Enterprise Vault recommends configuring the policy as mentioned below.

   {

       "Version": "2012-10-17",

       "Statement": [

           {

               "Action": [

                   "s3:Get*",

                   "s3:ListBucket"

               ],

               "Effect": "Allow",

               "Resource": [

                   "arn:aws:s3:::<source bucket>",

                   "arn:aws:s3:::<source bucket>/*"

               ]

           },

           {

               "Action": [

                   "s3:ReplicateObject",

                   "s3:ReplicateDelete",

                   "s3:ReplicateTags",

                   "s3:GetObjectVersionTagging"

               ],

               "Effect": "Allow",

               "Resource": "arn:aws:s3:::<destination bucket>/*"

           }

       ]

   }

   Run the following command from the computer where you have installed AWS CLI or AWS PowerShell to access the replication policy configured in the above step.
   aws s3api get-bucket-replication --bucket <sourcebucket name> --profile <aws cli configure name>
   After the command is executed, you will get the following output. You must save this JSON content in a file:
    

   {

       "ReplicationConfiguration": {

           "Role": "arn:aws:iam::XXXXXXXXXXX8:role/service-role/<role name>",

           "Rules": [

               {

                   "ID": "source-destination-replicate",

                   "Priority": 1,

                   "Filter": {},

                   "Status": "Enabled",

                   "Destination": {

                       "Bucket": "arn:aws:s3:::<destination bucket>"

                   },

   "DeleteMarkerReplication": {

                       "Status": "Disabled"

                   }

               }

           ]

       }

   }

    

5. By default, AWS uses the S3 replication behavior V2  version which does not replicate delete markers from the S3 source bucket. Therefore, change the S3 replication behavior version to V1, which replicates the delete markers.
   Edit the above saved JSON file as below:

   {

       "Role": "arn:aws:iam::XXXXXXXXXXX8:role/service-role/<role name>",

       "Rules": [

           {

               "ID": "source-destination-replicate",

               "Prefix": "",

               "Status": "Enabled",

               "Destination": {

                   "Bucket": "arn:aws:s3:::<destination bucket>"

               }

           }

       ]

   Here, the tags ReplicationConfiguration, "Filter": {}"Priority": 1 and DeleteMarkerReplication are removed, and "Prefix": "" is added.
6. Apply the updated JSON file to the AWS S3 source bucket using the following command.
   aws s3api put-bucket-replication --bucket <source bucket> --replication-configuration file://replication.json
7. Perform the delete operation on the AWS S3 source bucket, and verify if the delete markers are replicated on the destination bucket.