Skip to main content

Why Use a Collector

A collector is a virtual machine on your own network that receives logs locally and uploads them to the platform over HTTPS. Most sources do not need one: API, HEC, webhook, cloud-storage and event-stream integrations all run between the vendor's cloud and the platform, and a device that can send syslog over TLS can reach the cloud syslog endpoint directly. A collector earns its place when something about the source or the network rules that out.

When a Collector Is the Right Choice​

The source can only send in the clear. Many devices can send syslog only over UDP, or over TCP without TLS. Neither should cross the internet: anyone on the path can read the logs, and UDP can be spoofed. Sent to a collector on the same network instead, the cleartext leg stays inside your network and the leg across the internet is HTTPS.

The source can only use UDP. UDP has no acknowledgement and no retransmission. A datagram lost to congestion or a dropped packet is simply gone, and nothing on either side reports it — the loss grows with the distance and the number of networks in between. A collector on the local network keeps the UDP leg as short as possible, and forwards over a connection that does not lose data.

The source has no route to the internet. Devices on management networks, in segregated environments, or on sites with no direct internet access can reach a collector on the local network when they cannot reach the cloud.

The data only exists inside your network. An on-premises directory cannot be polled from the cloud. LDAP synchronisation runs from a collector for that reason, and NetFlow is received by one.

There are many senders at one site. Pointing a fleet of servers or network devices at one local address is simpler to manage than pointing each at the internet: one firewall rule to the internet instead of many, one place to change if the destination ever moves, and per-listener Customer Tags to attribute a whole site's records at once.

The upload path is unreliable. A collector that cannot reach the platform keeps receiving, and queues what it cannot yet upload; its Management tab shows the queue. When the connection returns, the backlog is delivered. Its capacity to ride out an outage is bounded by its disk, which is a reason to size it generously — see Fluency Collector for the recommended size.

You want sources authenticated. A device sending syslog directly to any listener identifies itself only by its address. A collector authenticates itself to the platform with its own token, so what it uploads is attributable to a known, registered appliance.

When a Collector Is Not Needed​

  • Cloud services — API, HEC, webhook, cloud storage and event streams. These run between clouds; a collector adds nothing.
  • Devices that can send syslog over TLS and can reach the internet. Send them straight to the cloud endpoint over Syslog TLS (or TLS RFC 6587, if that is the framing they use). See Syslog.
  • A single low-volume device where the cloud endpoint is reachable and TLS is available.

Costs to Weigh​

A collector is infrastructure you run: a VM to patch and monitor, a disk that can fill, a clock that has to stay synchronised, and a single point through which every source behind it flows — if it stops, they all stop together. Plan for monitoring its status on Platform → Administration → Collectors, and for what happens to the sources behind it during maintenance.

  • Fluency Collector — installing and registering a collector.
  • Managing Collectors — its listeners, network view and health.
  • Syslog — choosing between the cloud endpoint and a collector, and between protocols.