Skip to main content

AWS S3 with SQS Notifications

A delivery path, not a template

S3 with SQS notification is not an application template. It is the delivery mechanism several templates use: AWS CloudTrail, CloudWatch LogGroup and AWS EKS Logs each ask for an SQS notification queue url, and consume the bucket through it. This page describes the AWS-side wiring those templates depend on.

Many AWS services, and tools such as Fluent Bit, deliver their logs as files written to an S3 bucket. Fluency collects those files as they arrive:

  1. A source writes a file to the bucket.
  2. The bucket sends an "object created" event notification to an SQS queue.
  3. Fluency reads the queue with the role (or access key) it has been given.
  4. Fluency downloads the file the message points to, then deletes the message.

The queue is what makes this reliable. Fluency learns about every new file the moment it lands, in any order, without having to list the bucket.

Setting it up​

The wiring is the same for every source:

  • a queue the bucket can send to
  • a notification on the bucket
  • permission for Fluency to read the queue and the files

Access to the account itself, a registered role, comes first. See Connecting to AWS.

IngextS3SqsNotification.yaml does all three against an existing bucket:

  • It creates the queue.
  • It adds the notification without disturbing the bucket's existing ones.
  • It attaches the read permissions to your role.

Its parameters and outputs are described in Connecting to AWS, step 3. The QueueURL output is the value the templates ask for.

An older template, https://fluency-cloudformation.s3.us-east-2.amazonaws.com/FluencyPlatformS3AddNotification.yaml, remains available. It creates only the queue and its access policy. The bucket's event notification and the read permissions still have to be added by hand, as below.

By hand​

  1. Create the queue. In the SQS console, create a Standard queue. Give it an access policy that lets the bucket send to it:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": "*",
          "Action": "sqs:SendMessage",
          "Resource": "arn:aws:sqs:*:*:<queue-name>",
          "Condition": {
            "ArnEquals": {
              "aws:SourceArn": "arn:aws:s3:::<bucket-name>"
            }
          }
        }
      ]
    }

  2. Add the notification. In the S3 console:

    1. Open the bucket → Properties → Event notifications → Create event notification.
    2. Tick All object create events.
    3. Optionally, set a prefix.
    4. Choose the queue as the SQS queue destination.

  3. Grant read access. Attach a policy to the role, or to an IAM user for the access-key alternative. It needs two grants:

    • sqs:ReceiveMessage, sqs:DeleteMessage and sqs:GetQueueUrl on the queue.
    • s3:GetObject on the bucket's objects, narrowed to the prefix if you set one.

The manual tab of AWS CloudTrail walks through the same three steps with screenshots and the full read policy.

Constraints​

  • Same region. An S3 bucket can only notify a queue in its own region. Create the queue, and choose the template's AWS Region, to match the bucket.
  • One consumer per bucket. Use a bucket that doesn't already send its object-created notifications to another consumer, such as another SIEM.
  • Matching prefixes. If you scope the notification to a prefix, scope the read permission to the same one. A notification for files Fluency can't read produces messages but no events.
  • One queue per source. Several sources in the same account can share one role, but each gets its own queue. In CloudFormation, that means a distinct SQSQueueName for each.
  • No KMS-encrypted files. The read permissions don't include decrypting with a KMS key, so files encrypted with SSE-KMS can't be read.

Other data in S3​

The three templates cover CloudTrail, CloudWatch Logs exported to S3, and Fluent Bit output. Other data in S3 can be collected through the same queue-and-notification setup, but there is no application template for it, so the pipeline has to be configured by hand. Examples are load balancer, WAF and S3 access logs, or your own JSON and CSV files.

Reading a bucket without a queue​

Fluency can also read a bucket directly, without SQS. This is supported but not recommended. S3 lists a bucket's objects only in lexicographic order of their names, never by time, so direct reading keeps up with new data only when files are named to sort in the order they were written, for example with a timestamp at the front of the key. With a queue, file names don't matter.