Getting Data into Fluency

While Fluency support will help with the setting up of data collection, it is good to know how data is collected.

Typically, the Fluency team will schedule a 1-hour call to work on the ingress of all your data. This requires, that the admin for the various products be available to assist in this collaborative effort. For Windows servers, we use NXLog agents, for Firewalls and other Cloud based apps the admin will typically point the log data to Fluency. In other cloud app cases such as O365, API Keys will need to be generated and added – a very quick process.

The objective of Fluency is to get all the network and log data into the cloud. This relies on making a list of what audit logs you have and how that data is handled.

It is helpful to make a list of sources prior to installation. Common log sources are:

  • Network devices, such as a router, ie. Peplink, Cisco Meraki, or pfSense

  • Network security devices, such as a firewall, IPS, or Security Information Event Manager (SIEM)

  • Windows Active Directory / DHCP / DNS servers

  • Cloud services, such as Office365, G-Suite, CrowdStrike, and OpenDNS.

Additionally, network traffic data can also be collected:

  • Direct network traffic capture (local hardware/vm only, via SPAN or Mirrored port)

  • Netflow export (v5, v9, v10 or IPFIX)

Four basic methods of how log data are collected:

  1. Direct Syslog Ingress (can be syslog ssl): Fluency may provide either a VM or ISO image for an onsite device that can collect data via syslog and forward to the cloud, or one may turn on Syslog capabilities in the cloud. The format of the data is up to the user. This is Fluency’s preferred and most common method to ingress and evaluate the data, providing a more robust format that will enhance correlation.

  2. Windows Syslog Agent: Windows does not provide a clear mechanism to send logs remotely. Fluency can create a custom agent that will collect and forward Windows logs (AD/LDAP) via syslog. This is common in the industry. If a customer uses Windows AD, Fluency requests that the user implement this solution.

  3. API Polling: The system can reach out and collect data, most commonly via RESTful API connections. This is how SentinelOne, Office365, CrowdStrike and AWS data is collected. Fluency has a robust "Cloud Plugin" design that allows us ingress from an extensive list of supported cloud data sources. For local services that require polling, a local VM or ISO is needed.

  4. Webhook: Fluency also has the ability to generate a URL to accept webhook callbacks. This integration is used in our Bi-directional PagerDuty integration, and also allows use to collect events from Zoom Video

  5. Custom Agent: In the rare case that these techniques are unable to collect the data, Fluency can implement a custom GoLang Agent. GoLang can run on most operating systems. The agent can monitor a file and forward the data back to the cloud via an HTTPS/SSL connection.

Network data collection:

  1. Direct flow capture: (On-site devices only) Fluency may provide either a VM or ISO image for an onsite device that can collect network traffic via port SPAN or Mirroring and forward to the cloud

  2. Netflow Ingress: Fluency has the ability to accept Netflow traffic (normally port 2055). Supported formats are v5, v9, or v10 (IPFIX)

Historically, these mechanisms have been fairly encompassing. The ability to get good logs is important to Fluency. For example, in the VPN case, Fluency collects incoming connection information, VPN log information and outgoing information. Using the VPN log, Fluency can correlate which incoming connection is associated with what outgoing connections (one to many relationship) and assign the use associated with the connection. In this method, any IDS/IPS or network alerts can be associated with a user.

Source

What is Collected

Reason

Security Device Logs

The complete alert information from the security device.

Provides key events for alerting and supporting information for analysis.

Network Logs

Network flow information.

Network activity provides insight into what communication could be involved in an issue but did not trigger an alert.

Application Flows

Protocol exchange information above the network level.

Provides the best understanding of what the communications intent is.

DNS

Complete DNS exchange to include errors and recursive lookups.

Today’s networks often multi-home services. Host name is needed to understand when end point responded.

Files

Complete http file transfers.

Allows for network antivirus review in case endpoint does not have AV or is out of date.

Active Directory

Window events

Provides additional data such as user and asset. PCI requires a number of AD events to be compliance.

Office365 Logs

The complete Office365 record.

Not all the fields of an Office365 log can be accessed via its web interface. Storing in Fluency allows grouping, searching and analyzing the data faster than what is possible in the Office365 interface.

DHCP Logs

Asset requesting data.

Allows for the tracking of an asset and user, even when IP addresses are changed due to DHCP.

SMTP Logs

Header information and sending communication.

Provides a means to track how a particular email entered the system.

Asset Logs

Information from devices such as Infoblox, BlueCat and Forescout.

Allows for the tracking of asset and comparison of use by device.

SSL Certificates

SSL certificate used in communication.

Provides insight into the application and use even when communication is encrypted.

SIEM Data

Events and alerts.

Provides the capability of included data already collected without changing infrastructure.

Fluency is designed to collect and correlate logs from system, datalink and application levels (data source types). This enables better log management, especially for organizations that have made significant network investments. Information needed to investigate issues, whether security or network, are spread throughout systems and hidden in communications. Fluency collects, normalizes and correlates this information in order to change the job of administration from searching into leveraging critical data.

Fluency’s completeness mean that It provides the foundation for PCI and HIPAA audit requirements. Fluency centrally manages event and log information, maintains that data, and provides real-time access. Its log collection is one of the most complete in the industry.