I deployed several auto scaling setups before realizing: I had no idea when things went wrong! Instances could be failing, applications crashing, and I wouldn’t know until I manually checked.<\/p>
That’s when I built this comprehensive CloudWatch monitoring system. Now I get email alerts before users notice problems.<\/p>
A complete monitoring solution with:<\/p>
CloudWatch Monitoring Stack\n\u2502\n\u251c\u2500\u2500 ASG Metrics\n\u2502 \u251c\u2500\u2500 CPU Utilization Alarm (>80% = scale out + alert)\n\u2502 \u251c\u2500\u2500 Target Tracking (50% CPU target)\n\u2502 \u2514\u2500\u2500 Target Tracking (10 requests\/instance target)\n\u2502\n\u251c\u2500\u2500 ALB Metrics\n\u2502 \u251c\u2500\u2500 HTTP 4xx Errors (>5 in 6 min = alert)\n\u2502 \u251c\u2500\u2500 Target Response Time\n\u2502 \u2514\u2500\u2500 Healthy\/Unhealthy Host Count\n\u2502\n\u251c\u2500\u2500 Synthetics Canary\n\u2502 \u251c\u2500\u2500 Website availability check (every 1 minute)\n\u2502 \u251c\u2500\u2500 Success threshold (90%)\n\u2502 \u251c\u2500\u2500 Screenshots on failure\n\u2502 \u2514\u2500\u2500 Alerts when down\n\u2502\n\u251c\u2500\u2500 CIS Compliance Alarms\n\u2502 \u251c\u2500\u2500 Unauthorized API Calls\n\u2502 \u251c\u2500\u2500 Root Account Usage\n\u2502 \u251c\u2500\u2500 IAM Policy Changes\n\u2502 \u251c\u2500\u2500 CloudTrail Configuration Changes\n\u2502 \u251c\u2500\u2500 Security Group Changes\n\u2502 \u251c\u2500\u2500 Network ACL Changes\n\u2502 \u251c\u2500\u2500 Internet Gateway Changes\n\u2502 \u251c\u2500\u2500 Route Table Changes\n\u2502 \u2514\u2500\u2500 (many more security checks)\n\u2502\n\u2514\u2500\u2500 SNS Topic \u2192 Email Notifications\n \u251c\u2500\u2500 Scaling events\n \u251c\u2500\u2500 Alarm triggers\n \u251c\u2500\u2500 Canary failures\n \u2514\u2500\u2500 Compliance violations<\/code><\/pre>Technical Deep Dive<\/h3>1. CloudWatch ASG High CPU Alarm with Scaling<\/h3>
This is different from target tracking – this alarm triggers IMMEDIATE scale-out when CPU spikes:<\/p>
# Simple Scaling Policy - immediate scale out\nresource \"aws_autoscaling_policy\" \"high_cpu\" {\n name = \"high-cpu\"\n scaling_adjustment = 4 # Add 4 instances immediately\n adjustment_type = \"ChangeInCapacity\"\n cooldown = 300 # Wait 5 minutes before next scale\n autoscaling_group_name = aws_autoscaling_group.my_asg.name\n}\n\n# CloudWatch Alarm that triggers the policy\nresource \"aws_cloudwatch_metric_alarm\" \"app1_asg_cwa_cpu\" {\n alarm_name = \"App1-ASG-CWA-CPUUtilization\"\n comparison_operator = \"GreaterThanOrEqualToThreshold\"\n evaluation_periods = \"2\" # Check 2 times\n metric_name = \"CPUUtilization\"\n namespace = \"AWS\/EC2\"\n period = \"120\" # Each period is 2 minutes\n statistic = \"Average\"\n threshold = \"80\" # Trigger if above 80%\n\n dimensions = {\n AutoScalingGroupName = aws_autoscaling_group.my_asg.name\n }\n\n alarm_description = \"Triggers ASG scale-out if CPU above 80%\"\n\n ok_actions = [aws_sns_topic.myasg_sns_topic.arn]\n alarm_actions = [\n aws_autoscaling_policy.high_cpu.arn, # Execute scaling policy\n aws_sns_topic.myasg_sns_topic.arn # Send email\n ]\n}<\/code><\/pre>How this alarm works:<\/p>
Period 1 (minute 0-2):<\/p>
Samples: 75%, 82%, 79%, 85%, 80%, 77%<\/p>
Average: 79.7% \u2192 BELOW threshold (evaluation 1 = false)<\/p>
Period 2 (minute 2-4):<\/p>
Samples: 85%, 88%, 90%, 87%, 92%, 89%<\/p>
Average: 88.5% \u2192 ABOVE threshold (evaluation 2 = true)<\/p>
After 2 consecutive periods above 80%:<\/p>
- Alarm state changes to “ALARM”<\/li>
- Triggers scaling policy \u2192 Adds 4 instances<\/li>
- Sends SNS notification \u2192 Email sent<\/li>
- Enters cooldown for 300 seconds (prevents thrashing)<\/li><\/ol>
evaluation_periods = “2”:<\/p>
evaluation_periods = \"2\"<\/code><\/pre>Requires 2 consecutive high-CPU periods before alarming. This prevents false alarms from temporary spikes:<\/p>
# With evaluation_periods = 1 (too sensitive)\nCPU spike to 85% for 30 seconds \u2192 ALARM!\nCPU returns to 40% \u2192 Too late, already scaled\n\n# With evaluation_periods = 2 (better)\nCPU spike to 85% for 30 seconds \u2192 Wait...\nCPU stays high for another 2 minutes \u2192 Real problem, ALARM!\nCPU returns to 40% after spike \u2192 No alarm triggered<\/code><\/pre>period = “120”:<\/p>
Each evaluation period is 120 seconds (2 minutes). CloudWatch collects all CPU samples during this time and calculates average.<\/p>
alarm_actions vs ok_actions:<\/p>
alarm_actions = [...] # What to do when alarm fires\nok_actions = [...] # What to do when alarm recovers<\/code><\/pre>Example notification flow:<\/p>
t=0: CPU high \u2192 Alarm fires \u2192 alarm_actions execute\nt=5: Still high \u2192 No action (already alarmed)\nt=10: CPU normal \u2192 ok_actions execute \u2192 \"All clear\" email<\/code><\/pre>2. Simple Scaling vs Target Tracking<\/h3>
I use BOTH in my setup because they serve different purposes:<\/p>
Simple Scaling (immediate response):<\/p>
resource \"aws_autoscaling_policy\" \"high_cpu\" {\n scaling_adjustment = 4\n adjustment_type = \"ChangeInCapacity\"\n cooldown = 300\n}<\/code><\/pre>Behavior:<\/p>
- Alarm fires \u2192 Add exactly 4 instances immediately<\/li>
- Fixed response (always 4, never more, never less)<\/li>
- Cooldown prevents rapid scaling<\/li>
- Used for: Sudden traffic spikes, urgent situations<\/li><\/ul>
Target Tracking (gradual optimization):<\/p>
resource \"aws_autoscaling_policy\" \"avg_cpu_policy_greater_than_xx\" {\n policy_type = \"TargetTrackingScaling\"\n estimated_instance_warmup = 180\n\n target_tracking_configuration {\n predefined_metric_specification {\n predefined_metric_type = \"ASGAverageCPUUtilization\"\n }\n target_value = 50.0\n }\n}<\/code><\/pre>Behavior:<\/p>
- ASG continuously adjusts capacity to maintain 50% CPU<\/li>
- Smart calculations: if CPU is 70%, adds just enough instances to reach 50%<\/li>
- No cooldown, uses warmup instead<\/li>
- Used for: Normal operations, cost optimization<\/li><\/ul>
Why use both?<\/p>
Normal traffic (CPU 40-60%):<\/p>
\u2192 Target tracking handles this (adds\/removes 1-2 instances)<\/p>
Sudden spike (CPU jumps to 90%):<\/p>
\u2192 Simple scaling alarm fires (adds 4 instances immediately)<\/p>
\u2192 Target tracking continues adjusting after cooldown<\/p>
Result: Fast response to emergencies + efficient long-term scaling<\/p>
3. ALB Request Count Target Tracking<\/h3>
This is my favorite scaling metric – scale based on actual traffic!<\/p>
resource \"aws_autoscaling_policy\" \"alb_target_requests_greater_than_yy\" {\n name = \"alb-target-requests-greater-than-yy\"\n policy_type = \"TargetTrackingScaling\"\n autoscaling_group_name = aws_autoscaling_group.my_asg.id\n estimated_instance_warmup = 120\n\n target_tracking_configuration {\n predefined_metric_specification {\n predefined_metric_type = \"ALBRequestCountPerTarget\"\n resource_label = \"${module.alb.arn_suffix}\/${module.alb.target_groups[\"mytg1\"].arn_suffix}\"\n }\n target_value = 10.0 # 10 requests per instance\n }\n}<\/code><\/pre>resource_label explained:<\/p>
This is the tricky part! You need to identify your specific target group:<\/p>
resource_label = \"${module.alb.arn_suffix}\/${module.alb.target_groups[\"mytg1\"].arn_suffix}\"<\/code><\/pre>What this looks like:<\/p>
app\/sap-dev-alb\/1234567890abcdef\/targetgroup\/mytg1-\/0987654321fedcba\n \u2514\u2500\u2500 ALB ARN suffix \u2500\u2500\u2500\u2518 \u2514\u2500 Target Group ARN suffix \u2500\u2500\u2518<\/code><\/pre>Why this metric is better than CPU for web apps:<\/p>
CPU-based scaling:<\/p>
Light requests (static pages) \u2192 Low CPU \u2192 Doesn’t scale<\/p>
Heavy requests (database queries) \u2192 High CPU \u2192 Scales<\/p>
Problem: Number of concurrent users doesn’t match CPU!<\/p>
Request-based scaling:<\/p>
100 requests\/sec with 5 instances = 20 req\/instance \u2192 Scale out!<\/p>
100 requests\/sec with 10 instances = 10 req\/instance \u2192 Perfect!<\/p>
100 requests\/sec with 15 instances = 6.7 req\/instance \u2192 Scale in!<\/p>
Result: Scales with actual traffic volume<\/p>
Calculating target_value:<\/p>
target_value = 10.0<\/code><\/pre>How I chose 10:<\/p>
- Load test my application: 1 instance handles 30 requests\/sec comfortably<\/li>
- My application serves 3 requests\/sec on average<\/li>
- Target: 30 \/ 3 = 10 requests per instance<\/li>
- This keeps instances at ~33% capacity (room for bursts)<\/li><\/ol>
For your application:<\/p>
Max requests per instance = 50\nAverage response time = 100ms\nConcurrent requests capacity = 50 * (1 \/ 0.1) = 500\n\nSafe target = 500 * 0.7 = 350 requests per instance (70% capacity)<\/code><\/pre>How ASG calculates scale decisions:<\/p>
Current state:<\/p>
- 3 instances<\/li>
- 45 total requests\/sec<\/li>
- 45 \/ 3 = 15 requests per instance<\/li><\/ul>
Target: 10 requests per instance<\/p>
ASG math:<\/p>
Needed instances = 45 \/ 10 = 4.5 \u2192 Round up to 5\nAdd instances = 5 - 3 = 2\n\nAction: Scale out by 2 instances<\/code><\/pre>4. ALB 4xx Error Monitoring<\/h3>
Monitor application errors reported by the load balancer:<\/p>
resource \"aws_cloudwatch_metric_alarm\" \"alb_4xx_errors\" {\n alarm_name = \"App1-ALB-HTTP-4xx-errors\"\n comparison_operator = \"GreaterThanThreshold\"\n datapoints_to_alarm = \"2\" # 2 out of 3 periods must breach\n evaluation_periods = \"3\" # Check 3 periods\n metric_name = \"HTTPCode_Target_4XX_Count\"\n namespace = \"AWS\/ApplicationELB\"\n period = \"120\" # Each period is 2 minutes\n statistic = \"Sum\"\n threshold = \"5\" # More than 5 errors\n treat_missing_data = \"missing\"\n\n dimensions = {\n LoadBalancer = module.alb.arn_suffix\n }\n\n alarm_description = \"ALB 4xx errors above threshold\"\n ok_actions = [aws_sns_topic.myasg_sns_topic.arn]\n alarm_actions = [aws_sns_topic.myasg_sns_topic.arn]\n}<\/code><\/pre>datapoints_to_alarm = “2” with evaluation_periods = “3”:<\/p>
This means “2 out of last 3 periods must exceed threshold”:<\/p>
Period 1: 3 errors (< 5) \u2717<\/p>
Period 2: 7 errors (> 5) \u2713<\/p>
Period 3: 8 errors (> 5) \u2713<\/p>
Result: 2 out of 3 exceeded \u2192 ALARM<\/p>
Vs just evaluation_periods = “2”:<\/p>
Period 1: 7 errors (> 5) \u2713<\/p>
Period 2: 3 errors (< 5) \u2717<\/p>
Result: Only 1 exceeded \u2192 No alarm<\/p>
This prevents single-period anomalies from causing false alarms.<\/p>
statistic = “Sum” not “Average”:<\/p>
statistic = \"Sum\"<\/code><\/pre>For error counts, we want total errors, not average:<\/p>
Sum:<\/p>
Samples: 2, 0, 3, 1, 0, 2\nTotal: 8 errors in period \u2192 Compare to threshold<\/code><\/pre>Average:<\/p>
Samples: 2, 0, 3, 1, 0, 2\nAverage: 1.33 errors \u2192 Meaningless for threshold!<\/code><\/pre>treat_missing_data = “missing”:<\/p>
treat_missing_data = \"missing\"<\/code><\/pre>Options:<\/p>
- “missing” – Don’t count missing data (use for error counts)<\/li>
- “notBreaching” – Treat as not exceeding threshold (use for availability)<\/li>
- “breaching” – Treat as exceeding threshold (use for critical metrics)<\/li>
- “ignore” – Maintain current state<\/li><\/ul>
Example with error monitoring:<\/p>
Period 1: 10 errors \u2192 ALARM\nPeriod 2: No data (app down?) \u2192 \"missing\" keeps ALARM state\nPeriod 3: 12 errors \u2192 Still ALARM<\/code><\/pre>vs “notBreaching”:<\/p>
Period 1: 10 errors \u2192 ALARM\nPeriod 2: No data \u2192 Treated as 0 errors \u2192 OK\nPeriod 3: 12 errors \u2192 ALARM<\/code><\/pre>Missing = app might be down, keep alarming!<\/p>
NotBreaching = assumes no errors, false OK!<\/p>
5. CloudWatch Synthetics Canary – Website Monitoring<\/h3>
This is like having a robot continuously visit your website:<\/p>
# IAM Policy for Canary\nresource \"aws_iam_policy\" \"cw_canary_iam_policy\" {\n name = \"cw-canary-iam-policy\"\n policy = jsonencode({\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": \"cloudwatch:PutMetricData\",\n \"Resource\": \"*\",\n \"Condition\": {\n \"StringEquals\": {\n \"cloudwatch:namespace\": \"CloudWatchSynthetics\"\n }\n }\n },\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:PutObject\",\n \"logs:CreateLogStream\",\n \"logs:CreateLogGroup\",\n \"logs:PutLogEvents\",\n \"xray:PutTraceSegments\"\n ],\n \"Resource\": \"*\"\n }\n ]\n })\n}<\/code><\/pre># IAM Role\nresource \"aws_iam_role\" \"cw_canary_iam_role\" {\n name = \"cw-canary-iam-role\"\n assume_role_policy = jsonencode({\n \"Version\": \"2012-10-17\",\n \"Statement\": [{\n \"Effect\": \"Allow\",\n \"Principal\": {\"Service\": \"lambda.amazonaws.com\"},\n \"Action\": \"sts:AssumeRole\"\n }]\n })\n managed_policy_arns = [aws_iam_policy.cw_canary_iam_policy.arn]\n}<\/code><\/pre># S3 Bucket for Screenshots\nresource \"aws_s3_bucket\" \"cw_canary_bucket\" {\n bucket = \"cw-canary-bucket-${random_pet.this.id}\"\n force_destroy = true\n}<\/code><\/pre># Canary\nresource \"aws_synthetics_canary\" \"sswebsite2\" {\n name = \"sswebsite2\"\n artifact_s3_location = \"s3:\/\/${aws_s3_bucket.cw_canary_bucket.id}\/sswebsite2\"\n execution_role_arn = aws_iam_role.cw_canary_iam_role.arn\n handler = \"sswebsite2.handler\"\n zip_file = \"sswebsite2\/sswebsite2v1.zip\"\n runtime_version = \"syn-nodejs-puppeteer-6.0\"\n start_canary = true\n\n run_config {\n active_tracing = true\n memory_in_mb = 960\n timeout_in_seconds = 60\n }\n\n schedule {\n expression = \"rate(1 minute)\"\n }\n}<\/code><\/pre>What the canary does:<\/p>
Every minute:<\/p>
- Launches headless Chrome browser (Puppeteer)<\/li>
- Navigates to website (e.g., https:\/\/stacksimplify.com)<\/li>
- Waits for page to load<\/li>
- Takes screenshot<\/li>
- Uploads screenshot to S3<\/li>
- Reports success\/failure metric to CloudWatch<\/li>
- If failure, keeps screenshot for debugging<\/li><\/ol>
Canary script breakdown:<\/p>
const urls = ['https:\/\/stacksimplify.com'];\n\n\/\/ Navigate to URL\nconst response = await page.goto(url, {\n waitUntil: ['domcontentloaded'],\n timeout: 30000\n});\n\n\/\/ Check response\nif (response.status() < 200 || response.status() > 299) {\n throw `Failed to load url: ${url} ${response.status()}`;\n}\n\n\/\/ Wait for page to render\nawait page.waitFor(15000);\n\n\/\/ Take screenshot\nawait synthetics.takeScreenshot(stepName, 'loaded');<\/code><\/pre>runtime_version = “syn-nodejs-puppeteer-6.0”:<\/p>
This is the Node.js + Puppeteer runtime. Versions:<\/p>
- syn-nodejs-puppeteer-3.1 (deprecated)<\/li>
- syn-nodejs-puppeteer-6.0 (current)<\/li>
- syn-python-selenium-1.0 (Python alternative)<\/li><\/ul>
schedule.expression = “rate(1 minute)”:<\/p>
Options:<\/p>
rate(1 minute) # Every minute\nrate(5 minutes) # Every 5 minutes\nrate(1 hour) # Hourly\nrate(1 day) # Daily\n\ncron(0 9 * * ? *) # Every day at 9 AM UTC\ncron(0\/15 * * * ? *) # Every 15 minutes<\/code><\/pre>Cost calculation:<\/p>
rate(1 minute) = 60 checks\/hour * 24 hours * 30 days = 43,200 checks\/month\nPrice: $0.0012 per check\nCost: 43,200 * $0.0012 = $51.84\/month\n\nrate(5 minutes) = 8,640 checks\/month = $10.37\/month\nrate(15 minutes) = 2,880 checks\/month = $3.46\/month<\/code><\/pre>6. Synthetics Canary Alarm<\/h3>
Monitor canary success rate:<\/p>
resource \"aws_cloudwatch_metric_alarm\" \"synthetics_alarm_app1\" {\n alarm_name = \"Synthetics-Alarm-App1\"\n comparison_operator = \"LessThanThreshold\"\n datapoints_to_alarm = \"1\"\n evaluation_periods = \"1\"\n metric_name = \"SuccessPercent\"\n namespace = \"CloudWatchSynthetics\"\n period = \"300\"\n statistic = \"Average\"\n threshold = \"90\"\n treat_missing_data = \"breaching\"\n\n dimensions = {\n CanaryName = aws_synthetics_canary.sswebsite2.id\n }\n\n alarm_description = \"Website availability below 90%\"\n ok_actions = [aws_sns_topic.myasg_sns_topic.arn]\n alarm_actions = [aws_sns_topic.myasg_sns_topic.arn]\n}<\/code><\/pre>threshold = “90” with period = “300”:<\/p>
In 5-minute window (300 seconds):<\/p>
- Canary runs every 1 minute = 5 checks<\/li>
- Threshold = 90% = At least 4.5 successes required<\/li>
- 5 out of 5 successes = 100% \u2713<\/li>
- 4 out of 5 successes = 80% \u2192 ALARM!<\/li><\/ul>
treat_missing_data = “breaching”:<\/p>
If canary fails to run (Lambda issue, IAM problem):<\/p>
- Treat as failure<\/li>
- Trigger alarm<\/li>
- Send notification<\/li><\/ul>
This is correct for availability monitoring – not running = not available!<\/p>
7. CIS Compliance Alarms<\/h3>
This monitors AWS security best practices automatically:<\/p>
# Create Log Group for CIS\nresource \"aws_cloudwatch_log_group\" \"cis_log_group\" {\n name = \"cis-log-group-${random_pet.this.id}\"\n}\n\n# CIS Alarms Module\nmodule \"all_cis_alarms\" {\n source = \"terraform-aws-modules\/cloudwatch\/aws\/\/modules\/cis-alarms\"\n version = \"5.6.0\"\n\n disabled_controls = [\"DisableOrDeleteCMK\", \"VPCChanges\"]\n\n log_group_name = aws_cloudwatch_log_group.cis_log_group.name\n alarm_actions = [aws_sns_topic.myasg_sns_topic.arn]\n tags = local.common_tags\n}<\/code><\/pre>What CIS alarms monitor:<\/p>
The module creates alarms for:<\/p>
- UnauthorizedAPICalls – Detects failed API calls (might be attack)<\/li>
- NoMFARootAccountLogin – Root user without MFA<\/li>
- RootAccountUsage – Any root account activity<\/li>
- IAMPolicyChanges – Changes to IAM policies<\/li>
- CloudTrailConfigChanges – CloudTrail modifications<\/li>
- ConsoleSignInFailures – Failed console logins<\/li>
- DisableOrDeleteCMK – KMS key deletion<\/li>
- S3BucketPolicyChanges – S3 security changes<\/li>
- AWSConfigChanges – Config rule changes<\/li>
- SecurityGroupChanges – Security group modifications<\/li>
- NetworkACLChanges – NACL modifications<\/li>
- NetworkGatewayChanges – IGW\/VGW changes<\/li>
- RouteTableChanges – Route modifications<\/li>
- VPCChanges – VPC configuration changes<\/li><\/ol>
disabled_controls:<\/p>
disabled_controls = [\"DisableOrDeleteCMK\", \"VPCChanges\"]<\/code><\/pre>I disabled these because:<\/p>
- DisableOrDeleteCMK – I’m not using KMS in this demo<\/li>
- VPCChanges – Terraform makes frequent VPC changes during testing<\/li><\/ul>
For production, enable everything!<\/p>
How CIS alarms work:<\/p>
They use metric filters on CloudTrail logs:<\/p>
CloudTrail logs \u2192 CloudWatch Logs \u2192 Metric Filter \u2192 CloudWatch Alarm<\/code><\/pre>Example: Root Account Usage<\/p>
Filter pattern: {$.userIdentity.type=\"Root\" && $.userIdentity.invokedBy NOT EXISTS}\nWhen matched: Increment metric\nIf metric > 0: Alarm fires<\/code><\/pre>You get email like:<\/p>
Subject: ALARM: RootAccountUsage\n\nAlarmName: RootAccountUsage\nNewStateValue: ALARM\nStateChangeTime: 2025-01-15T14:30:00.000Z\n\nRoot account was used for API call:\n- Event: ConsoleLogin\n- Source IP: 203.0.113.10\n- Time: 2025-01-15T14:29:45Z\n\nAction required: Investigate root account usage<\/code><\/pre>8. SNS Topic for All Notifications<\/h3>
Central SNS topic receives all alerts:<\/p>
# Create SNS Topic\nresource \"aws_sns_topic\" \"myasg_sns_topic\" {\n name = \"myasg-sns-topic-${random_pet.this.id}\"\n}\n\n# Subscribe Email\nresource \"aws_sns_topic_subscription\" \"myasg_sns_topic_subscription\" {\n topic_arn = aws_sns_topic.myasg_sns_topic.arn\n protocol = \"email\"\n endpoint = \"rezachegini1994@gmail.com\"\n}\n\n# ASG Notifications\nresource \"aws_autoscaling_notification\" \"myasg_notifications\" {\n group_names = [aws_autoscaling_group.my_asg.id]\n notifications = [\n \"autoscaling:EC2_INSTANCE_LAUNCH\",\n \"autoscaling:EC2_INSTANCE_TERMINATE\",\n \"autoscaling:EC2_INSTANCE_LAUNCH_ERROR\",\n \"autoscaling:EC2_INSTANCE_TERMINATE_ERROR\",\n ]\n topic_arn = aws_sns_topic.myasg_sns_topic.arn\n}<\/code><\/pre>What sends to this SNS topic:<\/p>
- Auto Scaling Group:<\/li>
- CloudWatch Alarms:<\/li>
- Recovery Actions:<\/li><\/ol>
Email notification examples:<\/p>
ASG Launch:<\/p>
Subject: AUTO SCALING: Launch Successful\n\nGroup: myasg_12345\nCause: Target tracking increased desired capacity\nTime: 2025-01-15 10:30:00\nInstance: i-0abcd1234<\/code><\/pre>CPU Alarm:<\/p>
Subject: ALARM: App1-ASG-CWA-CPUUtilization\n\nState: ALARM\nThreshold: 80%\nValue: 87.5%\nTime: 2025-01-15 14:22:00\n\nScaling policy executed: Added 4 instances<\/code><\/pre>Canary Failure:<\/p>
Subject: ALARM: Synthetics-Alarm-App1\n\nState: ALARM\nSuccess Rate: 60% (below 90% threshold)\nFailed Checks: 2 out of 5\n\nScreenshots available in S3:\ns3:\/\/cw-canary-bucket-123\/sswebsite2\/2025-01-15T14-45-00\/<\/code><\/pre>Real-World Monitoring Scenarios<\/h3>Scenario 1: Traffic Spike Handling<\/h3>
What happens:<\/p>
14:00 - Normal traffic: 20 requests\/sec, CPU 45%\n14:05 - Marketing sends email blast\n14:06 - Traffic: 100 requests\/sec\n14:07 - Request-based scaling triggers: 20 req\/instance (target: 10)\n14:08 - ASG adds 5 instances (warmup 180s)\n14:11 - New instances healthy, traffic distributed\n14:12 - Requests\/instance back to 10 \u2713<\/code><\/pre>BUT:<\/p>
14:07 - CPU spikes to 92% (old instances overloaded)\n14:09 - High CPU alarm fires (>80% for 2 periods)\n14:10 - Simple scaling adds 4 more instances immediately\n14:13 - Total: 14 instances (2 initial + 5 tracking + 4 alarm)\n14:15 - CPU drops to 35%\n14:30 - Traffic decreases, target tracking scales in to 8 instances\n15:00 - Traffic normal, scaled back to 2 instances<\/code><\/pre>Email notifications received:<\/p>
14:09 - ALARM: High CPU\n14:10 - ASG Launch (4 instances)\n14:13 - OK: High CPU recovered\n14:30 - ASG Terminate (6 instances)<\/code><\/pre>Scenario 2: Application Error Detection<\/h3>
What happens:<\/p>
09:00 - Deploy new code version\n09:05 - ALB starts seeing 404 errors (broken endpoint)\n09:06 - Period 1: 8x 404 errors (> 5 threshold) \u2713\n09:08 - Period 2: 12x 404 errors (> 5 threshold) \u2713\n09:09 - Alarm fires (2 out of 3 periods breached)\n09:09 - Email sent with error count\n09:15 - I rollback deployment\n09:18 - Errors stop\n09:20 - Period 1: 0 errors \u2717\n09:22 - Period 2: 0 errors \u2717\n09:24 - Alarm OK, recovery email sent<\/code><\/pre>Scenario 3: Website Down Detection<\/h3>
What happens:<\/p>
11:00 - Canary check #1: Success (100%)\n11:01 - Canary check #2: Success (100%)\n11:02 - DNS misconfiguration breaks website\n11:02 - Canary check #3: FAILURE (timeout)\n11:03 - Canary check #4: FAILURE\n11:04 - Canary check #5: FAILURE\n11:05 - Success rate: 40% (2 success \/ 5 checks)\n11:05 - Alarm fires (below 90% threshold)\n11:05 - Email sent: \"Website down - 60% failure rate\"\n11:05 - Screenshots saved to S3 showing error page\n11:10 - I fix DNS\n11:12 - Canary check succeeds\n11:16 - Success rate back to 100%\n11:17 - Alarm OK, recovery email sent<\/code><\/pre>Scenario 4: Security Incident<\/h3>
What happens:<\/p>
16:30 - Attacker tries to delete security group\n16:30 - CloudTrail logs API call\n16:30 - CIS alarm \"SecurityGroupChanges\" metric filter matches\n16:30 - Metric incremented\n16:31 - Alarm evaluates, finds metric > 0\n16:31 - Alarm fires\n16:31 - Email sent:\n \"Security Group change detected\n User: unknown-user\n Action: DeleteSecurityGroup\n Security Group: sg-12345\n Source IP: 203.0.113.50\n Result: AccessDenied\"\n16:35 - I verify attacker was blocked by IAM\n16:35 - I add IP to blacklist<\/code><\/pre>What I Learned<\/h3>- Monitoring is not optional – Without CloudWatch, I was flying blind<\/li>
- Multiple scaling methods work together – Target tracking + simple scaling + scheduled<\/li>
- Request count is better than CPU – For web applications<\/li>
- CIS alarms are essential – Security monitoring automated<\/li>
- Synthetics catch problems early – Before users complain<\/li>
- SNS centralizes alerts – One topic, many sources<\/li>
- Alarm tuning is critical – Too sensitive = alert fatigue, too loose = miss issues<\/li><\/ol>","protected":false},"excerpt":{"rendered":"
Why we need Monitoring I deployed several auto scaling setups before realizing: I had no idea when things went wrong! Instances could…<\/p>\n","protected":false},"author":2,"featured_media":142,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"4","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/aws-monitoring-auto-scaling-terraform-reza-chegini-cphrc\/","footnotes":""},"categories":[85,10],"tags":[90,60,91,8,19,87],"series":[86],"class_list":["post-140","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","category-devops","tag-auto-scaling","tag-aws","tag-cloudwatch","tag-devops","tag-monitoring","tag-terraform","series-aws-with-terraform"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/aws-monitoring-auto-scaling-terraform-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/140","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=140"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/140\/revisions"}],"predecessor-version":[{"id":141,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/140\/revisions\/141"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/142"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=140"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=140"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=140"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=140"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}