Skip to content

korniichuk/cloudwatch-alarm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

51 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cloudwatch-alarm

Name: cloudwatch-alarm
Description: Amazon CloudWatch Alarm 2.0. CloudWatch to Slack smart notification
GitHub: https://github.com/korniichuk/cloudwatch-alarm

Table of Contents

Introduction

Amazon CloudWatch Alarm 2.0

amazon_cloudwatch_alarm_smart.png

Standard Amazon CloudWatch Alarm

amazon_cloudwatch_alarm_standard.png

How it works

how_it_works.jpg

Requirements

Please, install Python packages:

$ pip install -t . -r requirements.txt

Python lib versions

Create Slack webhook

Navigate to https://<your-team-domain>.slack.com/apps, like https://example.slack.com/apps. Search for and select Incoming WebHooks. Click Add Configuration button:

slack_-_add_configuration.png

Choose the default channel where messages will be sent (like #example) and click Add Incoming WebHooks integration. Copy and save the webhook URL (like https://hooks.slack.com/services/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) from the setup instructions.

Note: You need admin rights to your Slack.

Note: Slack marks public channels with # (e.g. #public) and private channels without # (e.g. private).

Create Amazon SNS topic

Create new Amazon SNS topic: https://docs.aws.amazon.com/sns/latest/dg/sns-getting-started.html#CreateTopic. Like example topic with arn:aws:sns:eu-west-1:539199393808:example ARN.

sns_-_topic_details.png

Create Amazon CloudWatch Alarm

Navigate to https://console.aws.amazon.com/cloudwatch/ and create new Amazon CloudWatch Alarm. Click Select metric button:

cloudwatch_-_select_metric.png

Select metric, like EC2 -> Per-Instance Metrics -> CPUUtilization:

cloudwatch_-_cpuutilization.png

Choose Source tab and select Image API checkbox. Copy and save json code. Click Select metric button:

cloudwatch_-_image_api.png

Saved json code example:

{
    "metrics": [
        [ "AWS/EC2", "CPUUtilization", "InstanceId", "i-0d91fdf2dbb765977", { "period": 300, "stat": "Average" } ]
    ],
    "title": "The percentage of CPU utilization",
    "start": "-PT3H",
    "end": "P0D",
    "timezone": "+0100"
}

Set up Alarm details. Enter Name (e.g. CPUUtilization) and Description (e.g. The percentage of CPU utilization). Specify the alarm condition. For example: select >=, enter 80, and enter 3 datapoints.

cloudwatch_-_alarm_details.png

Connect CloudWatch Alarm to created SNS topic and finally click Create Alarm button: cloudwatch_-_actions.png

AWS docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html

Create Amazon S3 bucket

Navigate to https://console.aws.amazon.com/s3/ and create new S3 bucket (e.g. example). Create new lifecyle rule for bucket (e.g. DeleteTmpAfter72h) with tmp/ prefix filter:

s3_-_lifecycle_rule.png

Configure expiration as below. Expire after 3 days. Permanently delete after 3 days. Clean up incomplete multipart uploads after 1 day.

s3_-_configure_expiration

Edit public access settings to your S3 bucket. Select your S3 bucket (e.g. example) and click Edit public access settings button. Deselect all checkboxes and click Save button:

s3_-_edit_public_access_settings.png

Enter confirm to confirm field and click Confirm button:

s3_-_confirm_public_access_settings.png

Create Amazon Lambda function

Create function base from blueprint

Navigate to https://console.aws.amazon.com/lambda/. Click Create function. Click Blueprints. Search for and select cloudwatch-alarm-to-slack-python3. Click Configure button:

lambda_-_blueprints.png

Enter function name (e.g. example) to Name field. In Role select Create a new role from one or more templates. In Role name field enter role name (e.g. example).

lambda_-_basic_information.png

Go to SNS trigger section and select your SNS topic. For example: example topic with arn:aws:sns:eu-west-1:539199393808:example ARN. Select the Enable trigger checkbox.

lambda_-_sns_trigger.png

Create KMS Key or use an existing KMS Key. For example: arn:aws:kms:eu-west-1:539199393808:key/4c3126f2-f3bf-453e-b32d-58c9356d84b7.

Go to Lambda function code section. Go to Encryption configuration subsection and select the Enable helpers for encryption in transit checkbox. Select AWS KMS key to encrypt in transit (e.g. arn:aws:kms:eu-west-1:539199393808:key/4c3126f2-f3bf-453e-b32d-58c9356d84b7).

lambda_-_encryption_configuration.png

Paste Slack channel into the slackChannel environment variable, like example. The Slack channel does not contain private info, so do NOT click Encrypt button.

Paste Slack webhook URL into the kmsEncryptedHookUrl environment variable. You must exclude the protocol from the URL (e.g. hooks.slack.com/services/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). Click Encrypt button. Finally click Create function button:

lambda_-_environment_variables.png

Upgrade function

Clone cloudwatch-alarm repository:

$ git clone https://github.com/korniichuk/cloudwatch-alarm.git

Modify cloudwatch-alarm/lambda_function.py file. Replace confluence_url and cloudwatch_url var values. For example, from:

confluence_url = 'http://www.korniichuk.com'
cloudwatch_url = 'http://www.korniichuk.com'

to:

confluence_url = 'https://bit.ly/1d3LdqJ'
cloudwatch_url = 'https://amzn.to/2NeZooo'

Update metric var value with saved Image API json. For example, from:

metric = """{
    "metrics": [[
        "LogMetrics",
        "MetricName",
        {"period": 3600, "stat": "Sum"}]],
    "title": "Title",
    "start": "-P1D",
    "end": "P0D",
    "timezone": "+0100"}"""

to:

metric = """{
    "metrics": [[
        "AWS/EC2",
        "CPUUtilization",
        "InstanceId",
        "i-0d91fdf2dbb765977",
        {"period": 300, "stat": "Average"}]],
    "title": "The percentage of CPU utilization",
    "start": "-PT3H",
    "end": "P0D",
    "timezone": "+0100"}"""

Docs: Metric widget structure and syntax

Replace bucket_name var value to your bucket name. For example, from:

bucket_name = 'korniichuk'

to:

bucket_name = 'example'

Navigate to cloned cloudwatch-alarm repository and install Python packages:

$ cd cloudwatch-alarm
$ pip install -t . -r requirements.txt

Prepare zip.zip with lambda_function.py file and Python dependencies:

$ rm -r *.dist-info __pycache__
$ zip -r zip.zip .

Navigate to your lambda function (e.g. example). Go to Function code section and select Upload a .zip file from Code entry type dropdown. Click Upload button. Upload zip.zip file. Finally click Save button:

lambda_-_upload_function_package.png

Note: We use boto3 ver. 1.9.41 and botocore ver. 1.12.41 because default Lambda's packages do not support get_metric_widget_image function.

Edit AWS IAM role

Navigate to https://console.aws.amazon.com/iam/. Select your role (e.g. example). Select Permissions tab. Click Attach policies button:

iam_-_permissions.png

Search for and select AmazonS3FullAccess and CloudWatchFullAccess. Click Attach policy button. See final result below.

iam_-_summary.png

Note: Do not use *FullAccess roles on production. You can grant access to concrete S3 bucket (e.g. example). You can precisize CloudWatchFullAccess policy to cloudwatch:* plus logs:* only.

Test Amazon Lambda function

Navigate to your lambda function (e.g. example). Click Select a test event.. and select Configure test events:

lambda_-_configure_test_events.png

Enter event name to Event name field (e.g. Example). Copy json code from test-event.json file to code area or enter your own. Click Create button:

lambda_-_create_new_test_event.png

Select created event (e.g. Example) and click Test button:

lambda_-_run_test.png

Note: You can maximize CPU utilization with $ dd if=/dev/zero of=/dev/null command.

Change Slack message retention

Navigate to Slack channel. Click Channel Settings icon. Select Edit message retention...:

slack_-_channel_settings.png

Change default message retention to 3 Days. Click Save button:

slack_-_retention_settings.png