Amazon Data Lifecycle Manager to automate the creation, retention, and deletion of Amazon EBS volume snapshots. Instead of creating snapshots manually and deleting them in the same way (or building a tool to do it for you), you simply create a policy, indicating (via tags) which volumes are to be snapshotted, set a retention model, fill in a few other details, and let Data Lifecycle Manager do the rest. Data Lifecycle Manager is powered by tags, so you should start by setting up a clear and comprehensive tagging model for your organization.
It turns out that many of our customers have invested in tools to automate the creation of snapshots, but have skimped on the retention and deletion. Sooner or later they receive a surprisingly large AWS bill and find that their scripts are not working as expected. The Data Lifecycle Manager should help them to save money and to be able to rest assured that their snapshots are being managed as expected.
Creating and Using a Lifecycle Policy
Data Lifecycle Manager uses lifecycle policies to figure out when to run, which volumes to snapshot, and how long to keep the snapshots around. You can create the policies in the AWS Management Console, from the AWS Command Line Interface (CLI) or via the Data Lifecycle Manager APIs; I’ll use the Console today. Here are my EBS volumes, all suitably tagged with a department:
I access the Lifecycle Manager from the Elastic Block Store section of the menu:
Then I click Create Snapshot Lifecycle Policy to proceed:
Then I create my first policy:
I use tags to specify the volumes that the policy applies to. If I specify multiple tags, then the policy applies to volumes that have any of the tags:
I can create snapshots at 12 or 24 hour intervals, and I can specify the desired snapshot time. Snapshot creation will start no more than an hour after this time, with completion based on the size of the volume and the degree of change since the last snapshot.
I can use the built-in default IAM role or I can create one of my own. If I use my own role, I need to enable the EC2 snapshot operations and all of the DLM (Data Lifecycle Manager) operations.
Newly created snapshots will be tagged with the aws:dlm:lifecycle-policy-id and aws:dlm:lifecycle-schedule-name automatically; I can also specify up to 50 additional key/value pairs for each policy:
I can see all of my policies at a glance:
I took a short break and came back to find that the first set of snapshots had been created, as expected (I configured the console to show the two tags created on the snapshots):
Things to Know
Here are a couple of things to keep in mind when you start to use Data Lifecycle Manager to automate your snapshot management:
Data Consistency – Snapshots will contain the data from all completed I/O operations, also known as crash consistent.
Pricing – You can create and use Data Lifecycle Manager policies at no charge, you pay the usual storage charges for the EBS snapshots that it creates.
Availability – Data Lifecycle Manager is available in the US East (N. Virginia), US West (Oregon), and Europe (Ireland) Regions.
Tags and Policies – If a volume has more than one tag and the tags match multiple policies, each policy will create a separate snapshot and both policies will govern the retention. No two policies can specify the same key/value pair for a tag.
Programmatic Access – You can create and manage policies programmatically! Take a look at the CreateLifecyclePolicy, GetLifecyclePolicies, and UpdateLifeCyclePolicy functions to get started. You can also write an AWS Lambda function in response to the createSnapshot event.
Error Handling – Data Lifecycle Manager generates a “DLM Policy State Change” event if a policy enters the error state.
In the Works – As you might have guessed from the name, we plan to add support for additional AWS data sources over time. We also plan to support policies that will let you do weekly and monthly snapshots, and also expect to give you additional scheduling flexibility.
Summary
Create a lifecycle policy in Amazon DLM:
1. In the Amazon Elastic Compute Cloud (Amazon EC2) console, under Elastic Block Store, select Lifecycle Manager.
2. Select Create Snapshot Lifecycle Policy.
3. Enter a Description for the policy.
4. Select a resource type. Select Volume if you want to schedule snapshots for volumes with a specific tag. Select Instance if you want to schedule snapshots for all volumes attached to an instance with a specific tag.
5. Select the tags associated with the Amazon EBS volume or Amazon EC2 instance, depending on the option chosen above.
6. Add a Schedule name to your policy. Any snapshot created with this policy is automatically tagged with the schedule name entered here.
7. Enter the number of hours that will elapse between policy runs.
8. Enter the policy run start time. Snapshot creation starts within one hour of the specified start time.
9. Set the Retention rule to the maximum number of snapshots that you want to retain. The oldest snapshot beyond this number will be deleted.
10. Optionally, select Copy Tags from volume to copy all user-defined tags on a source volume to snapshots of the volume created by this policy.
11. Optionally, add additional tags to the created snapshots by selecting Add Tag. These tags are in addition to any tags that were copied from the volumes or added as default by Amazon DLM.
12. Select an AWS Identity and Access Management (IAM) role that has snapshot create and delete permissions. The Default role has the required permissions. If you don’t want to use the default role, you can create a new role with the required permissions using the IAM console. Select the new role when creating your lifecycle policy. The following permissions are required for a role to use Amazon DLM.
"ec2:CreateSnapshot",
"ec2:CreateSnapshots",
"ec2:DeleteSnapshot",
"ec2:DescribeVolumes",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots"13. Review the Policy Summary to verify that the specified rules meet your requirements.
14. Select Enable policy to start the policy runs at the next scheduled time. If Enable policy is not selected, the policy will not create or delete snapshots.
15. Select Create Policy.
