Skip to main content

Getting Data into Fluency

While Fluency Support will help with setting up data collection, it is good to know how your data is collected.

Fluency Platform is the latest method of data ingestion into the Fluency SIEM.

As the front-end of the SIEM, Fluency Platform has the ability to integrate with and accept feeds from many devices and services by other vendors.

The objective here is to get all the network and audit data into your Fluency instance.

It is helpful to make a list of what audit logs you have and how that data is handled prior to installation. For the full list, see the Integration Matrix.

Common log sources are:

  • Network devices, such as firewalls and/or routers, ie. Fortigate, Cisco Meraki, SonicWall, or Peplink
  • Network security devices, such as an IPS, or another Security Information Event Manager (SIEM)
  • Windows Servers, typically the Active Directory/DHCP/DNS servers
  • Cloud services, such as Office365, G-Suite, or Duo
  • Network services: such as Cisco Umbrella (OpenDNS)
  • EDR services, such as Bitdefender, CrowdStrike, or SentinelOne

Basic methods of how data are collected:

  1. Direct Syslog Ingress (or Syslog w/TLS): Fluency can provide either a VM (OVA) or installation package for an on-prem deployment that can collect data via Syslog and upload it to the cloud. Alternatively, the customer can also send Syslog directly to the cloud (a Syslog endpoint URL/IP address is provided upon request). The format of the data is up to the user, with each 'record' spanning a single line of text. This is Fluency’s preferred and most common method to ingress and evaluate data.

  2. Windows Syslog Agent: Windows does not provide a native mechanism to send event logs remotely. Fluency makes use of the NxLog agent software that will collect and forward Windows Event Logs via Syslog. If a customer uses Windows AD (from a Domain Controller), Fluency requests that the user implement this solution. This practice is common in the industry.

  3. API Polling: Fluency can reach out and collect data, most commonly via RESTful API connections. This is how Office365/Microsoft365, SentinelOne, CrowdStrike and AWS data are collected. Fluency has a robust "Cloud Plugin" design that allows us to support an extensive list of data sources. For local/on-prem services that require API polling (such as LDAP integration), a local/on-prem VM or Collector is required.

  4. HTTPs Event Collector (HEC): Fluency also has the ability to generate a URL to HTTPs event streams. This allows us to collect events from any source that is compatible with Splunk's HTTP Event Collector (HEC).

  5. Custom Agent: In the rare case that these techniques are unable to collect the data, Fluency also has a custom Log Forwarder agent hat can be configured to read/follow logs from a particular folder, and then upload the contents to the Cloud via an HTTPs connection. Currently, this agent is installed via RPM, and is supported on CentOS/RHEL 7.

The ability to get good logs is important to Fluency, and historically, these mechanisms have been fairly encompassing.

Page last updated: 2024 Feb 27