A key component of monitoring (and observability) is the collection and processing of metrics. In the AWS ecosystem, these metrics are found in AWS Cloudwatch Metrics.
AWS CloudWatch Metrics was recently updated to make data collection even easier than before. Metric Streams replaces traditional polling methods to deliver large quantities of data directly to an AWS S3 bucket.
Prior to this new feature, CloudWatch Metrics were fetched through API polling.
Fluency’s integration with AWS Cloudwatch is capable of collecting events from Metric Streams, via an intermediary Amazon Kinesis Data Firehose and AWS S3 bucket.
Our integration will also deploy intemediate AWS SQS and IAM resources to facilitate data collection.
We will use the AssumeRole API to securely connect with your intermediary resources without the need to pass around IAM credentials. An IAM Policy w/ External ID will explicity define Fluency’s scope of access to your account.
At any time, you may remove the integration and/or intermediary resources to revoke access.
The integration consists of three sections:
The user will first initiate the integration on the Fluency Portal. Creating a new plugin from the Integration section will generate a unique External ID for use in the client AWS account.
Fluency provides AWS CloudFormation scripts for the user to create AWS resources for the integration. Executing the CloudFormation with the External ID, will create AWS resources, including an AWS SQS queue.
The final step will test the above AWS SQS queue in the Fluency interface. Once the test is successful, the user can complete the integration with the additon of Metric ingress rules.
Login to the Fluency Cloud portal: https://companyname.cloud.fluencysecurity.com.
Open the main dropdown menu and choose the Integrations option under the Data Ingress section.
On the following page, navigate to the Cloud Infrastructure as a Service section.
To Add an integration for AWS Monitoring, choose the AWS icon from the group on the left side of the page to create a new AWS integration endpoint.
NOTE: If an integration endpoint was setup previously, you can also select and modify it from the rigt side of the page.
In the pop-up window, provide an Account name for the integration. The value will be used within the Fluency interface only to distinguish the different integrations. It is suggested to avoid using spaces in this field.
Also, enter the AWS Account ID for your account. Choose the Save button to add the integration endpoint.
Select the AWS integration endpoint from the list on the right side of the page, in the same Cloud Infrastructure as a Service section. Choose the pencil icon to edit/configure the connector.
For the initial configuration, this will bring up a pop-up window, with the Account name and the External ID.
The External ID will be required in the following step on the AWS console.
After executing the CloudFormation script, you are given the opprtunity to test the configuration using the Test Connection button.
On subsequent visits for the plugin configuration page, select the Show External ID button in the upper left corner of the page.
AWS CloudFormation allows you to configure AWS resources from script/code. This makes deployment easy, and consistent, and greately decrease the possiblity os errors or misconfigurations.
Fluency provides a CloudFormation script to initialize the integration, and add one (1) Metric stream integration, via an AWS SQS queue.
Fluency also provides another “update” CloudFormation script to add additonal metric streams after the first one
The following resources will be created on AWS:
- 1 IAM Role
- 2 IAM Policy
- 1 SQS Queue
- 1 SQS Queue Policy
- 1 S3 Bucket
While the above script is free to use, keep in mind that AWS CloudFormation is a paid service, and you will incur a charge from AWS for using it.
Link to the CloudFormation file on S3 (initialization script):
Navigate to the CloudFormation section of the AWS Management Console.
Under the “Stacks” section, choose “Create stack” (with new resources, standard).
On the following page, specify a template, and choose the desired template (using Amazon S3 URL) from above:
Click “Next” to continue.
Give this deployment a name, and specify some parameters. You can choose your own name, or keep the default values provided by Fluency.
Click Next to continue.
Configure additional items (optional).
Click Next to continue.
Review the deployment, when complete, choose Create stack to deploy.
When the deployment is complete, return to the Fluency interface for the following step.
Return the Integrations page, and edit the AWS Monitoring configurations.
On the following pop-up, you are given the opprtunity to test the configuration using the Test Connection button.
Proceed to the next page, if the test completes successfully.
You may be sent to the edit page directly, on subsequent visits for the plugin configuration page, select the Show External ID button in the upper left corner of the page.
On the plugin configuration page, select the + Metric Stream button in the upper left of the Metric Streams tab.
In the pop-up window, select the appropriate AWS Region for the SQS queue obtained from the CloudFormation run. Enter the SQS queue URL.
You are also given the opprtunity to test the configuration here.
Select the Import Rules tab to see the list of metric import rules.
To add a new import rule, use the + New Rule button.
You will be able to choose from a list of pre-defined rules.
This completes the initial integration.
Page last updated: 2022 Nov 11