AWS defines S3 as "Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. Amazon S3 is designed for 99.999999999% (11 9's) of durability". In this article we'll be going through different storage tiers s3 provides and Lifecycle Configuration which will allow the objects to be transitioned automatically from one storage tier to another automatically according to defined rules.
So there are following Storage tier currently provided by AWS S3.
- S3 Standard
- S3 Intelligent Tiering
- S3 Infrequent Access(IA)
- S3 One Zone Infrequent Access(1Z IA)
- S3 Glacier
- S3 Glacier Deep Archieve
S3 Standard: S3 Standard offers high durability, availability, and performance object storage for frequently accessed data. Because it delivers low latency and high throughput, S3 Standard is appropriate for a wide variety of use cases, including cloud applications, dynamic websites, content distribution, mobile and gaming applications, and big data analytics. This storage tier is most expensive tier in terms of storage.
S3 Intelligent Tiering: It is recently introduced feature which lets us to automatically transition objects between standard tier and IA tier.
One major thing to note when setting the storage to Intelligent tier is make sure the object you're storing should at-least will be kept for 30 or more days since it charges minimum for 30 days. It is priced same as the standard tier.
S3 IA: S3 Standard-IA is for data that is accessed less frequently, but requires rapid access when needed. S3 Standard-IA offers the high durability, high throughput, and low latency of S3 Standard, with a low per GB storage price and per GB retrieval fee. This combination of low cost and high performance make S3 Standard-IA ideal for long-term storage, backups, and as a data store for disaster recovery files.
S3 1Z IA: This is very similar to S3 IA with one major difference that the stored object will only be kept in One Availability Zone and hence will only have 99.5% overall availability. This is 25% cheaper than S3 IA tier storage.
S3 Glacier and Glacier Deep Archieve: These are the lowest cost tier storage with same durability but a longer retrieval time resulting from 2 days to 12 days.
To know the complete pricing and features of those storage tiers, visit official documentation S3 Object Storage Classes
Now, lets talk about auto transition of objects from one storage tier to another. Sometimes you might have some sort of data say some data sheets in .csv format. These data sheets need to be accessed more often at the start but once done processing they are rarely accessed.
However these data sheets might be needed later.
So you now want to move those data sheets to another storage tier like Glacier so that you'll be charged less for the storage and also maintain the same durability as standard tier. But how ???
There's a feature provided by AWS S3 which lets you to automate the transition of objects from one storage tier to another called LifeCycle Management. We'll be setting up lifecycle management through CloudFormation however it is also very simple via AWS Console too. Feel free to choose any of the methods.
So, we'll be using serverless framework to quickly generate a template and deploy it to the CloudFormation.
Our serverless.yml file will be similar as :
service: s3lifecycle
provider:
name: aws
runtime: nodejs12.x
resources:
Resources:
myS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: test-bucket-lifecycle-rule
VersioningConfiguration:
Status: Enabled
LifecycleConfiguration:
Rules:
- Status: Enabled
NoncurrentVersionTransitions:
- TransitionInDays: 30
StorageClass: STANDARD_IA
- TransitionInDays: 120
StorageClass: GLACIER
Transitions:
- TransitionInDays: 100
StorageClass: STANDARD_IA
# Expiration:
# ExpiredObjectDeleteMarker: true
Id: TransitionRules
What we actually are doing is, we're defining a property to our S3 Bucket and note that Versioning is Enabled however you can opt to not enable the versioning but this way it will be more detailed that's why i enabled the versioning. Now we're a rule to Transition the objects to STANDARD_IA after 30 days of creation and again move it to GLACIER after 120 days of all the Non current version objects that is previous version. If you're not using Versioning then you simply have to replace NoncurrentVersionTransitions with Transitions . For current version we simply can use Transtions.
Also Note that Expiration is commented out, if removed comment it will let you expire objects after allocated Number of days.
Now to deploy in your terminal type:
sls deploy
it will be deployed and lifecycle rule will be set up in your defined bucket.
Thanks for Reading !!
