Data Normalization

Data Normalization is mapping the attribute-value pairings into a consistent naming and type convention.

Base Record

Field

Type

Description

c

String

Collector who produced the message

@type

String

The type of record this is: metadata or event

Flow Values

The base flow record is:

Field

Type

Description

start_ms

Integer

GMT in Linux Epoch time

dur

Integer

duration in milliseconds of the session length

proto

Integer

Protocol Number

sip

String

source address, often the source IP address

sp

Integer

Source port

dip

String

destination address, often the destinations IP address

dp

Integer

Destination port

rxP

Integer

Received number of packets

txP

Integer

Transmitted number of packets

rxB

Integer

Received number of bytes

txB

Integer

Transmitted number of bytes

rf

Integer

Mask of the combined received flags

totalB

Integer

Total of Bytes in session

partition

String

name of shard

dHost

String

Passive mapped name from DNS query

http

Object

HTTP protocol metadata

dns

Object

DNS protocol metadata

meta

Object

Device related metadata.

The base object often has a geo-ip lookup.

Field

Type

Description

s_country

String

Two letter Internet country code of source

s_city

String

City related to destination by IP address of source

s_org

String

Registered Organization owner of source

s_isp

String

Registered ISP of source

d_country

String

Two letter Internet country code of destination

d_city

String

City related to destination by IP address of destination

d_org

String

Registered Organization owner of destination

d_isp

String

Registered ISP of destination

The base object might have an associated user from LDAP logs.

Field

Type

Description

su

String

The domain name of the source's user

du

String

The domain name of the destination's user

DHCP Data

Field

Type

Description

hostname

String

Referenced to Hostname

mac

String

Network Interface Card Machine Address (data link level)

HTTP

Nested in the HTTP root attribute are the attributes for an HTTP connection.

Field

Type

Description

host

String

The HTTP host variable.

agent

String

the type of process that is running this request

referer

String

The URI that caused this page to be requested

xforward

String

If a firewall has X-Forward on, this field will show the address of the internal system. Fluency will generate a second record to correlate this activity from source to destination.

uris

Object

this is the URI object

The URI Object contains the following fields.

Field

Type

Description

cmd

String

The HTTP host variable.

uri

String

Uniform Resource Identifier, what people normally type in their browser after the site name.

status

Integer

Return status value

t

String

Type of content

mime

String

mime type

DNS

Field

Type

Description

id

Integer

The ID for the query provided by the client

query

Object

A query by a client

DNS query object

Field

Type

Description

flags

Array

Array of DNS Flags

questions

Array

Request by client for a lookup

Questions

Field

Type

Description

name

String

Name being requested. DNS names end in a period (.)

type

String

Record Type requested

class

String

DNS Class

Answers

Field

Type

Description

name

String

Name being requested. DNS names end in a period (.)

ttl

Integer

Time to Live. How many seconds this answer should be cached by the client.

type

String

Record Type requested

class

String

DNS Class

cname

String

Conical Name reference. When this is provided, the client should request an IP address from this system.

ipv4

String

The returning address is an IPv4 address

ipv6

Sting

The returning address is an IPv6 address

Event Values

The base flow record is:

Field

Type

Description

@message

String

The raw message unless it is in JSON messages, the event name.

@timestamp

Integer

GMT in Linux Epoch time

@level

String

System message level.

@source

String

Where the message was generated

@tags

Array

Array of information labels

@incidents

Array

Array of incident (issue) labels

@facility

String

Facility of the source

@sender

String

device that sent the alarm

@fields

Object

Parsed data of the message

AWS CloudTrail

Fluency downloads and populates the @field with the AWS CloudTrail JSON log. Learn more about CloudTrail here:

There are key fields to pay attention to:

Field

Type

Description

eventName

String

Action

userIdentity.username

String

Username of the action

sourceIPAddress

String

Source of the connection that made the API request

There is no defined list of eventNames.

Office 365

The values of the @fields attribute will contain the JSON record of the audit. Since this record is retrieved as a JSON object, the @message field will contain the @fields.Operation value.

An office list of the fields are here: https://docs.microsoft.com/en-us/office365/securitycompliance/detailed-properties-in-the-office-365-audit-log‚Äč

Descriptions are from the Office MS site when a definition existed.

Field

Type

Description

CreationTime

String

GMT String of the time

Id

String

The ID of the report entry. The ID uniquely identifies the report entry.

Operation

String

The name of the user or admin activity. The value of this property corresponds to the value that was selected in the Activities drop down list. If Show results for all activities was selected, the report will included entries for all user and admin activities for all services. For a description of the operations/activities that are logged in the Office 365 audit log, see the Audited activities tab in Search the audit log in the Office 365 Security & Compliance Center. For Exchange admin activity, this property identifies the name of the cmdlet that was run.

OrganizationId

String

The GUID for your Office 365 organization.

RecordType

Integer

The type of operation indicated by the record. The following values indicate the record type.

UserKey

String

An alternative ID for the user identified in the UserIDproperty. For example, this property is populated with the passport unique ID (PUID) for events performed by users in SharePoint. This property also might specify the same value as the UserID property for events occurring in other services and events performed by system accounts.

Version

Integer

Indicates the version number of the activity (identified by the Operation property) that's logged.

Workload

String

The Office 365 service where the activity occurred. The possible values for this property are: SharePoint OneDrive Exchange AzureActiveDirectory DataCenterSecurity Compliance Sway SecurityComplianceCenter PowerBI MicrosoftTeams ThreatIntelligence

ClientIP

String

The IP address of the device that was used when the activity was logged. The IP address is displayed in either an IPv4 or IPv6 address format.

ClientIPAddress

String

SharePoints version of ClientIP

ObjectId

String

For Exchange admin audit logging, the name of the object that was modified by the cmdlet. For SharePoint activity, the full URL path name of the file or folder accessed by a user. For Azure AD activity, the name of the user account that was modified.

UserId

String

The user who performed the action (specified in the Operation property) that resulted in the record being logged. Note that records for activity performed by system accounts (such as SHAREPOINT\system or NT AUTHORITY\SYSTEM) are also included in the audit log.

UserDomain

String

Identity information about the tenant organization of the user (actor) who performed the action.

CorrelationId

String

Unlisted Attribute. The attribute appears to relate a request with its response. Appears in Sharepoint logs when a search is followed by an FileUploaded.

EventSource

String

Identifies that an event occurred in SharePoint. Possible values are SharePoint and ObjectModel. (Sharepoint)

ExternalAccess

String

For Exchange admin activity, specifies whether the cmdlet was run by a user in your organization, by Microsoft datacenter personnel or a datacenter service account, or by a delegated administrator. The value False indicates that the cmdlet was run by someone in your organization. The value True indicates that the cmdlet was run by datacenter personnel, a datacenter service account, or a delegated administrator. For Exchange mailbox activity, specifies whether a mailbox was accessed by a user outside your organization. (Exchange)

ItemType

String

The type of object that was accessed or modified. Possible values include File, Folder, Web, Site, Tenant, and DocumentLibrary. (SharePoint)

ListId

String

Unlisted Attribute

ListItemUniqueId

String

Unlisted Attribute

Site

String

The GUID of the site where the file or folder accessed by the user is located. (Sharepoint)

UserAgent

String

Information about the user's browser. This information is provided by the browser. (Sharepoint)

WebId

String

Unlisted Attribute. Web Hash Identifier

SourceFileExtension

String

The MS DOS file extension used for application mapping

SiteUrl

String

The URL used to connect to this resource

SourceFileName

String

The filename by itself with extension

SourceRelativeUrl

String

The relative directory. This plus site, plus filename is the complete URL

Record Types

Value

Meaning

1

Indicates a record from the Exchange admin audit log.

2

Indicates a record from the Exchange mailbox audit log for an operation performed on a singled mailbox item.

3

Also indicates a record from the Exchange mailbox audit log. This record type indicates the operation was performed on multiple items in the source mailbox (such as moving multiple items to the Deleted Items folder or permanently deleting multiple items).

4

Indicates a site admin operation in SharePoint, such as an administrator or user assigning permissions to a site.

6

Indicates a file or folder-related operation in SharePoint, such as a user viewing or modifying a file.

8

Indicates an admin operation performed in Azure Active Directory.

9

Indicates OrgId logon events in Azure Active Directory. This record type is being deprecated.

10

Indicates security cmdlet events that were performed by Microsoft personnel in the data center.

11

Indicates Data loss protection (DLP) events in SharePoint.

12

Indicates Sway events.

14

Indicates sharing events in SharePoint.

15

Indicates Secure Token Service (STS) logon events in Azure Active Directory.

18

Indicates Security & Compliance Center events.

20

Indicates Power BI events.

22

Indicates Yammer events.

24

Indicates eDiscovery events. This record type indicates activities that were performed by running content searches and managing eDiscovery cases in the Security & Compliance Center. For more information, see Search for eDiscovery activities in the Office 365 audit log. 25, 26, or 27 - Indicates Microsoft Teams events.

25, 25 or 27

Indicates Microsoft Teams events.