Amazon Web Services (AWS)
Amazon Web Services (AWS)
Fluency collects from AWS in two ways:
- From S3. It reads the log files that AWS services deliver to an S3 bucket, and an SQS queue tells it when each new file lands.
- From AWS APIs. It polls the API directly.
Every AWS source needs access to your account first. Set that up once, as described in Connecting to AWS, and every source in this section reuses it.
For more detailed information regarding AWS usage, refer to the vendor's documentation.
Official AWS Documentation:
Connecting to AWS
There are two ways to give the platform read access to an AWS account:
- Assumed role — recommended. You create an IAM role in your account that trusts your Fluency tenant's own AWS identity, and register it under Settings → Advanced → AWS Roles. The platform reads with temporary credentials on that role, so no long-lived secret leaves your account. One role serves every AWS source on the account, and each source adds only the read permissions it needs. The Setup Role wizard on that page hands you the CloudFormation template, already filled in with your values.
- Access key pair — supported alternative. An IAM user's access key ID and secret, entered on the install form. Every AWS template accepts one, but it is a long-lived credential you have to rotate and retire yourself. Use it where a cross-account role is not an option. It is the only method for AWS GovCloud accounts.
The EC2 Instance Role method is no longer supported; use an assumed role instead.
Connecting to AWS walks through both methods.
What can be collected
Four application templates, in the AWS Services category of Platform → Applications → Install Application From Template, cover the AWS sources most accounts start with:
| Template | Collects | Delivered through | Credentials | Guide |
|---|---|---|---|---|
| AWS CloudTrail | CloudTrail events | S3 + SQS notification | Role, or key pair | AWS CloudTrail |
| CloudWatch LogGroup | CloudWatch Logs events, exported to S3 | S3 + SQS notification | Role, or key pair | Amazon CloudWatch |
| AWS EKS Logs | Container logs that Fluent Bit writes to S3 | S3 + SQS notification | Role, or key pair | EKS Logs |
| Amazon GuardDuty | GuardDuty findings | GuardDuty API, polled per region | Role, or key pair | AWS GuardDuty |
Each guide's Install the Application section walks through its form, and every form is also described field by field in Install Application From Template.
Other data in S3 can also be collected, through a pipeline configured by hand. See AWS S3 (with SQS).
In this section
Start here: Connecting to AWS sets up the role every AWS source uses.
- Connecting to AWS — the assumed role (recommended) and the access-key alternative.
- AWS CloudTrail — CloudTrail events via S3 and SQS, including creating the trail.
- Amazon CloudWatch — CloudWatch Logs, with CloudFormation templates for the Firehose, subscription-filter, Kinesis and Lambda paths.
- EKS Logs — container logs that Fluent Bit writes to S3.
- AWS S3 (with SQS) — the bucket-to-queue delivery the S3-based templates share, and collecting other data from S3.
- Amazon Data Firehose — delivery-stream ingestion (formerly Kinesis Data Firehose).
- Amazon VPC — the VPC integration overview.
- AWS GuardDuty — GuardDuty findings.
- AWS Security Lake — OCSF-normalized data from Security Lake.
Related
- Integration Matrix — which ingress method each product supports.
- Applications — the full catalog.
- Troubleshooting — no data arriving.