Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Getting Data into Fluency

Table of contents

While Fluency Support will help with setting up data collection, it is good to know how your data is collected.

Typically, the Fluency team will schedule a 1-hour call to work on the ingress of your data. We ask that the admin(s) for the various products be available to assist in this collaborative effort. For most on-premise appliances/devices, the admin(s) will typically configure the Syslog export to send to Fluency. For Windows Servers, we make use of a agent software (NXLog) to convert data to and export Syslog. Many Cloud-based Applications will also have settings for Syslog export. Other Cloud Applications will have API integration. The admin(s) will be asked to generate API token/keys or to perform OAuth authentication.

The objective of Fluency is to get all the network and log data into the cloud. This relies on making .

It is helpful to make a list of what audit logs you have and how that data is handled prior to installation. Common log sources are:

  • Network devices, such as a router, ie. Peplink, Cisco Meraki, or pfSense
  • Network security devices, such as a firewall, IPS, or Security Information Event Manager (SIEM)
  • Windows Active Directory / DHCP / DNS servers
  • Cloud services, such as Office365, G-Suite, CrowdStrike, SentinelOne, or OpenDNS

Additionally, network traffic data can also be collected:

  • Direct network traffic capture (on-prem hardware only, via SPAN or Mirrored port)
  • Netflow export (v5, v9, v10 or IPFIX)

Four basic methods of how log data are collected:

  1. Direct Syslog Ingress (or Syslog w/TLS): Fluency can provide either a VM (OVA) or installation package for an on-prem deployment that can collect data via Syslog and upload it to the cloud. Alternatively, the customer can also send Syslog directly to the cloud (a Syslog endpoint URL/IP address is provided upon request). The format of the data is up to the user, with each ‘record’ spanning a single line of text. This is Fluency’s preferred and most common method to ingress and evaluate data.
  2. Windows Syslog Agent: Windows does not provide a native mechanism to send event logs remotely. Fluency makes use of the NxLog agent software that will collect and forward Windows event logs (AD/LDAP) via Syslog. If a customer uses Windows AD, Fluency requests that the user implement this solution. This practice is common in the industry.
  3. API Polling: Fluency can reach out and collect data, most commonly via RESTful API connections. This is how SentinelOne, Office365, CrowdStrike and AWS data is collected. Fluency has a robust “Cloud Plugin” design that allows us ingress from an extensive list of supported cloud data sources. For local/on-prem services that require polling, a local/on-prem VM or Collector is required.
  4. Webhook: Fluency also has the ability to generate a URL to accept webhook callbacks. This integration is used in our Bi-directional PagerDuty integration, and also allows us to collect audit events from Zoom Video.
  5. Custom Agent: In the rare case that these techniques are unable to collect the data, Fluency also has a custom Log Forwarder agent hat can be configured to read/follow logs from a particular folder upload to the Cloud an HTTPS connection. Currently, this agent is installed via RPM, and is supported on CentOS/RHEL 7.

Network data collection:

  1. Direct flow capture: On-premise Fluency collectors can collect network traffic via port SPAN or Mirroring and upload to the cloud.
  2. Netflow Ingress: Fluency (all collector types) has the ability to accept Netflow traffic (normally on UDP port 2055). Supported formats are v5, v9, or v10 (IPFIX).

Historically, these mechanisms have been fairly encompassing. The ability to get good logs is important to Fluency. For example, in a Fluency deployment capturing data from a network containing a VPN device, Fluency collects incoming connection information, the VPN log information and outgoing connection information. Using the VPN logs, Fluency can correlate which incoming connection is associated with which outgoing connections (one to many relationship) and assign the user associated with the connection. Using this Fusion method, any IDS/IPS or network alerts can be associated with a user.

Fluency is designed to collect and correlate logs from system, datalink and application levels (data source types). This enables better log management, especially for organizations that have made significant network investments. Information needed to investigate issues, whether security or network, are spread throughout systems and hidden in communications. Fluency collects, normalizes and correlates this information in order to change the job of administration from searching into leveraging of critical data.

Fluency’s completeness mean that it provides the foundation for PCI and HIPAA audit requirements. Fluency centrally manages event and log information, maintains that data, and provides real-time access. Its log collection is one of the most complete in the industry.

Page last updated: 2021 Nov 24