Automate AWS CloudWatch Log Group Retention Using Lambda In Python

CloudWatch organises logs in a log group and when a new log group is created, it’s retention period is set to Never expire, which means logs will be retained forever. When we are streaming logs via cloudwatch agent using cloudwatch agent json file we dont get option to define log group retention in that json file also.

What Is Accomplished By This Automation?

When a new CloudWatch log group is created directly via console or via cloudwatch agent, a CloudWatch event rule triggers a lambda function. Then the lambda function sets a desirable retention time for the CloudWatch log group. After that retention time all log stream(s) data of log group will be deleted automatically.

Prerequisites

AWS account IAM user of that AWS account (It is recommended to perform task via IAM user, not from root account) IAM user should be authorised to access services for creating this automation task.

Create IAM Role

Here we are creating IAM role for an AWS service called Lambda. By using this role Lambda can access the other AWS resources.

Create an IAM role and add the following policy into that. To know how to create IAM role and attach policy for a service please refer to this document.

    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutRetentionPolicy",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Open the Functions page on the Lambda console.

1) Choose Create function.

2) Click on create function

3) Fill in basic information and choose our IAM role that we have created as displayed below

4) Once lambda function is created goto code editor and use below python code

import json
import boto3
import logging
import os

print('Loading function')
logger= logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info("Received event: " + json.dumps(event, indent=2))
    loggroupname=event['detail']['requestParameters']['logGroupName']
    logger.info(loggroupname)
    client = boto3.client('logs')
    response=client.put_retention_policy(
        logGroupName=loggroupname,
        retentionInDays=int(os.getenv('RetentionPeriod'))
    )
    logger.info(response)

Create CloudWatch Event Rule

Go to services → CloudWatch → Rules → click on Create rule .

Event Source → choose Event Pattern → select CloudWatch Logs in Service Name , AWS API Call via CloudTrail in Event Type (If you don’t have Trail setup in CloudTrail, do first. To get help refer this document), CreateLogGroup in Specific operation(s)

Targets → select Lambda function → select previously created lambda function → click on Configure details . Give Name, Description, State should be enabled → click on Create rule .

Above automation can be done for the infrastructure where retention policy for log group is not set while building it.

While building infrastructure this retention policy can also be set by adding below code in your build cloudformation template.

Resources:
    logGroup0:
        Type: AWS::Logs::LogGroup
        Properties:
            KmsKeyId: !Ref keyId
            LogGroupName: !Sub '${AWS::StackName}-lg'
            RetentionInDays: !Ref retainInDays

Note:- Make sure to set "KmsKeyId","LogGroupName","RetentionInDays" as per your infra build.

Conclusion

We have seen how to set CloudWatch log groups retention time automatically practially. Kindly note after deploying this automation into AWS account, it will only set retention time for newly created log groups only not the existing ones.
To update retention policy for existing log groups we have to use python script which you can find here