Skip to main content

Data Collection

Every source reaches Fluency through one of a small number of transports. The Integrations section has a guide per product — what to configure in the vendor's console, which template to install — and each of those guides rests on one of the transports described here. This section covers what they have in common: who connects to whom, what has to be exchanged, what happens when something stops, and how to tell that it is working.

Whatever the transport, the platform end is usually the same: an application installed from a template on Platform → Applications, which creates the data source for that transport, the parser for that product, and the index the records land in. The transport decides what the data source is, and what you have to set up on the other side.

The Transports​

TransportWho connectsWhat joins the two sidesTemplatesTypical sources
API pollingFluency → vendor, on a scheduleA credential you create in the vendor's consoleMost of the catalogSaaS and security platforms: Microsoft 365, Okta, CrowdStrike, Google Workspace
SyslogDevice → Fluency listenerAn address and port, plus a CA for TLSFirewalls, network devices, serversCisco ASA, FortiGate, Palo Alto, Linux, Windows via NXLog
HECVendor → FluencyA URL and token the install generatesBitdefender, Coro, Zscaler, and a generic passthroughAnything with a Splunk HEC output
WebhooksVendor → FluencyA URL the install generates, and a shared secretGitHubServices that post JSON on each event
Cloud storageFluency → your bucket and queueA role (or key pair) and an SQS queue URLAWS CloudTrail, CloudWatch, EKSServices that write log files to S3
Event streamsFluency → your streamA connection stringAzure Event HubsAzure platform and Entra ID logs

Two pieces of infrastructure sit alongside them:

  • Fluency Collector — a virtual machine on your own network. It receives syslog and NetFlow from devices that cannot reach the cloud securely, and runs the queries that only work from inside the network.
  • LDAP — directory synchronisation, which runs from a collector because the directory is not reachable from the cloud.

How They Differ​

Pull or push. API polling, cloud storage and event streams are pull: Fluency opens the connection, nothing on your side accepts one, and the source keeps the data until it is read. Syslog, HEC and webhooks are push: the source opens the connection as events happen, and whether anything is kept when the destination is unreachable is up to the source.

Which side goes first. For pull transports you create a credential in the source and paste it into the template. For HEC and webhooks the order reverses — the install generates the URL and token that the source then needs, so Fluency comes first. Syslog has no credential at all; the device is pointed at an address, and the template that parses it is best installed before the device starts sending.

What "Running" means. On a pull template, a Running badge means the platform is trying; it does not prove the credential works. On a push template it means the endpoint is listening; it says nothing about whether anything is sending. Every transport page says what to check instead, and Confirm Data Is Arriving covers the platform-side checks they share.

How long an outage can last. Pull sources wait for you only as long as the source retains data — an API window, a queue's or a stream's retention. Push sources may not wait at all:

TransportWhile collection is stoppedTime to fix it before data is lost
API pollingThe vendor keeps the data; the next poll resumes from its last positionThe vendor's API window — as little as 24 hours for some sources
Cloud storageFiles accumulate in the bucket; notifications wait in the queueThe queue's retention — four days by default
Event streamsEvents wait in the streamThe stream's retention — one day on Azure Event Hubs Basic
HEC and webhooksThe vendor's own retry behaviour, if anyVendor-dependent — from none at all to a short replay window
SyslogDevices do not store and resend; a collector queues what it cannot uploadNone for a device sending directly; the collector's disk when one is in between

The short windows are the ones worth watching most closely, with a check that does not depend on volume: a quiet tenant and a dead integration look the same on an ingress chart.

Choosing a Transport​

Most of the time the template decides: a product's guide says which transport its template uses, and that is the one to use. The choice is yours in a few cases:

  • A device can send syslog both over TLS and in the clear. Use TLS — to the cloud endpoint directly, or to a collector. See Choosing a Protocol.
  • A device can only send UDP or unencrypted TCP, or cannot reach the internet. Send it to a collector.
  • A product has no template but can send to Splunk. Use the HEC Input Passthrough template — the records are stored unparsed, but they are in.
  • A product has no template and no Splunk output. If it can send syslog, send it to a listener: records nothing parses are kept in the default index. Otherwise, see the product's page in Integrations, and the Integration Matrix for every product's path at a glance.

In This Section​

  • API Polling — credentials, cadence and backfill, API windows, rotation.
  • Syslog — the cloud endpoint and collector listeners, protocols and framing, TLS trust, and how templates claim records; with Ingress Management for watching which devices are sending.
  • HEC (Splunk-compatible) — the HTTP Event Collector endpoint.
  • Webhooks — vendor-posted JSON, signatures, and one-shot delivery.
  • Cloud Storage — S3 with SQS notifications, and replaying an archive from S3.
  • Event Streams — Azure Event Hubs, and how AWS streams get in.
  • Fluency Collector — installing, registering and managing the appliance.
  • LDAP (Requires collector) — directory synchronisation through a collector.