Skip to main content

Event Streams

A managed event stream sits between the services that produce logs and whatever reads them. Producers write to the stream; the stream holds each event for a fixed retention period; readers pull at their own pace and keep track of how far they have read. Fluency is one such reader. Like API polling and cloud storage, the connection runs outbound from Fluency, with a credential you grant it, and nothing on your network accepts a connection.

Azure Event Hubs is the stream with an application template, and the main way Azure platform logs reach the platform: Azure Monitor diagnostic settings on almost any Azure resource, and Microsoft Entra ID's own diagnostic settings, can stream to an event hub. See Azure Event Hubs for the Azure-side walkthrough. AWS's streaming services reach the platform by other routes — see AWS Streams.

How It Works​

  1. Producers write to the stream. In Azure, you point a resource's diagnostic setting at an event hub. The hub divides what it receives into partitions.
  2. The stream retains events for its configured retention period, whether or not anyone has read them.
  3. Fluency reads, through a consumer group — a named view of the hub that tracks its own position in each partition, independently of every other consumer group.
  4. Fluency records how far it has read — in a storage container you provide, if you provide one — and passes each event to the template's parser and index.

What You Supply​

The Azure Event Hubs template's parameters map onto those pieces:

ParameterValue
Event Hub endpointThe hub's Connection string–primary key, from a shared access policy with the Listen right only, created on the event hub itself. A policy created on the namespace gives a string without ;EntityPath=<hub name> at the end — and usually with far more rights than reading needs. The hub-level string is the one to use.
consumer group nameOptional. Left empty, the reader uses the hub's built-in $Default group.
connection string to access the storage account container and blob storage container for eventhub checkpointsOptional, together. An Azure Storage account and a container in it, where the reader keeps its position.
datalake index namePre-filled AzureEventHubs.

The connection string is a credential: it carries the policy's key. Unlike most vendor secrets, though, it can be read again from Shared access policies at any time.

Three Things That Decide How Robust It Is​

Retention​

An event that is not read within the hub's retention period is gone. That makes retention the length of outage the integration can survive — a stopped application, a regenerated key, a deleted policy. The tier sets the ceiling: Event Hubs Basic retains for one day, Standard for up to seven. One day leaves little time to notice and fix a problem over a weekend; choose Standard and raise retention for a hub whose contents matter.

Consumer Groups​

Each consumer group keeps one position per partition, and Azure expects one active reader per partition in a group. If anything else already reads the hub — another SIEM, an Azure Function, Stream Analytics — sharing $Default with it means the two interfere. Give Fluency a dedicated consumer group, created on the hub, and name it in consumer group name. The Basic tier allows only $Default, which is another reason to use Standard for a shared hub.

A consumer group name that does not exist on the hub is an error on Azure's side, and the application collects nothing.

Checkpoints​

Without checkpoint storage the reader has nowhere durable to keep its position, so a restart cannot resume exactly where it stopped, and re-reads what the hub still holds. With it, a restart picks up where the last one left off. For a small, single-purpose hub the difference is a few duplicates after a restart; for a busy hub with many partitions, set up checkpoint storage — a general-purpose storage account and one container is all it takes.

Confirming It Works​

The hub's Overview page in the Azure portal charts Incoming Messages and Outgoing Messages, and between them they localise most problems:

  • Incoming stays at zero — no producer is writing to this hub. Check the diagnostic settings point at this namespace and this hub; a diagnostic setting with no hub name creates its own hub in the namespace (Entra ID's audit logs default to one called insights-logs-audit), and the application is reading a different one. New diagnostic settings take around fifteen minutes to start streaming.
  • Incoming rises, Outgoing stays at zero — nothing is reading. Check the connection string ends in the right EntityPath, the consumer group exists, and the application is Running.
  • Both rise — Fluency is reading; look on the platform side, at Platform → Usage and the AzureEventHubs index. See Confirm Data Is Arriving.

Rotating the Key​

Every shared access policy has two keys, which is what makes an overlap possible:

  1. Uninstall and reinstall the application with the policy's Connection string–secondary key. (Parameters are read-only once installed, so a new value means a reinstall.)
  2. Once data is flowing again, regenerate the primary key in Azure.

The next rotation goes the other way round. Keep the outage between steps shorter than the hub's retention and nothing is lost; with checkpoint storage configured, the reinstalled application resumes from the stored position.

AWS Streams​

AWS's streaming services do not have a stream-reading template of their own. Each reaches the platform through another transport:

ServiceHow it gets in
Amazon Data FirehoseDeliver to an S3 bucket, and collect the bucket through its SQS notification queue — see Cloud Storage and Amazon Data Firehose.
CloudWatch LogsA subscription filter sends a log group to Firehose, which delivers to S3, which the CloudWatch LogGroup template reads — see Amazon CloudWatch.
Kinesis Data StreamsNo template. It can be read through a connector on Platform → Integrations; contact Fluency support before building one.