Leveraging CloudWatch Synthetics for Proactive Incident Response
Amazon CloudWatch Synthetics enables the creation of canaries — configurable scripts scheduled to run and monitor endpoints and APIs. These canaries mimic customer interactions, allowing continuous verification of the customer experience, even in the absence of actual customer traffic.
Canaries operate seamlessly over both HTTP and HTTPS protocols.
Creating Your First Canary
Let’s start by creating a basic canary. For instance, consider a scenario where I want to monitor the availability of my website https://www.adityanama.com/.
Important:
Use Synthetics canaries to monitor only the endpoints and APIs you own or have permissions for. Note that these canaries may increase traffic based on their frequency settings.
Create Canary:
- Navigate to “Application Signals” in the CloudWatch console, click on “Synthetics Canaries,” and select “Create canary.”
- Choose a blueprint, like “Heartbeat Monitor” and configure the required parameters.
A variety of canary blueprints cater to different monitoring needs:
Heartbeat Monitor:
- Monitors specified URLs, capturing screenshots and HTTP archive files (HAR).
- Allows detailed performance analysis of web pages.
- Supports monitoring multiple URLs with runtime version syn-nodejs-puppeteer-3.1 or later.
API Canary:
- Tests basic Read and Write functions of a REST API.
- Can make multiple API calls, supporting various API functionalities.
- Supports multi-step canaries with runtime version syn-nodejs-2.2 or later, allowing monitoring of multiple APIs in a single canary.
Broken Link Checker:
- Collects links within a specified URL, testing up to the specified number of links.
- Provides a report with checked links, status codes, failure reasons, and screenshots.
- Detects various link errors, including 404 Page Not Found and invalid URLs.
Visual Monitoring Blueprint:
- Compares screenshots during a canary run with a baseline run, triggering failure if the discrepancy exceeds a set threshold.
- Requires syn-puppeteer-node-3.2 or later runtime.
- Offers the ability to edit the baseline run, set new baselines, and ignore specific areas during visual comparisons.
Canary Recorder:
- Uses the CloudWatch Synthetics Recorder Chrome extension to record click and type actions on a website.
- Automatically generates a Node.js script for creating a canary to replicate recorded steps.
GUI Workflow Builder:
- Verifies actions on a webpage, such as form submissions.
- Allows specifying actions like Click, Verify Selector, Verify Text, Input Text, and Click with Navigation.
- Provides a detailed summary of executed steps for each canary run with certain runtimes.
Note: Examples in the blog will focus on the heartbeat monitoring blueprint for practical implementation and understanding.
Configure Schedule:
- Set the schedule for your canary to run at regular intervals (e.g., every 5 minutes).
Set Up Alarms:
Create CloudWatch Alarms based on Success Percentage, Duration or Failed threshold.
Interpreting Canary Results
- Evaluating CloudWatch Alarms Outcomes for Detecting Incidents
AWS CloudWatch Synthetics enhances incident response and threat detection by:
- Proactive Monitoring: Canaries simulate user interactions, proactively monitoring application health, and detecting potential issues.
- Alerts and Notifications: Configurable alarms trigger on canary-defined thresholds, notifying incident response teams for prompt investigation.
- Incident Triage: Canaries yield valuable data aiding incident triage, providing insights into issues detected during their execution.
- Threat Detection: While not explicit in design, canary-detected anomalies can signal potential security threats, like unauthorized access attempts.
Security Considerations for CloudWatch Synthetics Canaries:
Use Secure Connections:
- Employ encrypted connections (https://) to safeguard canary code and test run results, preventing exposure of sensitive information.
Canary Naming Considerations:
- Choose canary names that don’t disclose proprietary details.
- Point canaries only at controlled websites and endpoints.
Secrets in Canary Code:
- Avoid passing sensitive information directly into canary code.
- Store sensitive scripts in Amazon S3 and pass the S3 location instead of using a zip file.
Permissions Considerations:
- Restrict access to CloudWatch Synthetics resources.
- Implement tight permissions on S3 buckets storing test run results, logs, and screenshots.
Object Versioning for Canary Code:
- Utilize object versioning on S3 buckets containing canary code to ensure running the intended code version.
Stack Traces and Exception Messages:
- Capture and log exceptions but avoid including sensitive information.
- Securely handle URLs in errors and consider redacting restricted parameters.
Scope IAM Roles Narrowly:
- Configure canaries to visit trusted URLs to prevent potential exposure to malicious scripts.
- Run Lambda functions with IAM roles having limited permissions.
Sensitive Data Redaction:
- Redact sensitive information from logs and reports, including URLs, headers, and bodies.
- Use configurations like restrictedUrlParameters to control information exposure.
- Enable or restrict headers and request/response bodies based on sensitivity.
Conclusion:
To wrap up, we’ve uncovered the fundamentals of using Amazon CloudWatch Synthetics to proactively monitor applications through configurable canaries. These canaries, mimicking user interactions, provide a vigilant eye on application health even in the absence of actual user traffic.
Next Blog Sneak Peek:
In our upcoming blog, we’ll plunge into the realm of custom canaries. Get ready for hands-on insights, advanced configuration tips, and practical examples. Whether you’re new to CloudWatch Synthetics or seeking to refine your monitoring, the next blog will empower you to optimize custom canaries for a proactive incident response. Stay tuned!
References:
Six Habits of Highly Effective Threat Detection/Incident Response Teams
