Link Search Menu Expand Document

Normalization

Table of contents
  1. Base Record
  2. Flow Values
    1. DHCP Data
    2. HTTP
    3. DNS
      1. DNS query object
      2. Questions
      3. Answers
  3. Event Values
    1. AWS CloudTrail
    2. Office 365

Data Normalization is mapping the attribute-value pairings into a consistent naming and type convention.

Base Record

Field Type Description
@type String The type of record this is: metadata or event

Flow Values

The base flow record is:

Field Type Description
start_ms Integer GMT in Linux Epoch time
dur Integer duration in milliseconds of the session length
proto Integer Protocol Number
sip String source address, often the source IP address
sp Integer Source port
dip String destination address, often the destinations IP address
dp Integer Destination port
rxP Integer Received number of packets
txP Integer Transmitted number of packets
rxB Integer Received number of bytes
txB Integer Transmitted number of bytes
rf Integer Mask of the combined received flags
totalB Integer Total of Bytes in session
partition String name of shard
dHost String Passive mapped name from DNS query
http Object HTTP protocol metadata
dns Object DNS protocol metadata
meta Object Device related metadata.
c String Collector who produced the message

The base object often has a geo-ip lookup.

Field Type Description
s_country String Two letter Internet country code of source
s_city String City related to destination by IP address of source
s_org String Registered Organization owner of source
s_isp String Registered ISP of source
d_country String Two letter Internet country code of destination
d_city String City related to destination by IP address of destination
d_org String Registered Organization owner of destination
d_isp String Registered ISP of destination

The base object might have an associated user from LDAP logs.

Field Type Description
su String The domain name of the source’s user
du String The domain name of the destination’s user

DHCP Data

Field Type Description
hostname String Referenced to Hostname
mac String Network Interface Card Machine Address (data link level)

HTTP

Nested in the HTTP root attribute are the attributes for an HTTP connection.

Field Type Description
host String The HTTP host variable.
agent String the type of process that is running this request
referer String The URI that caused this page to be requested
xforward String If a firewall has X-Forward on, this field will show the address of the internal system. Fluency will generate a second record to correlate this activity from source to destination.
uris Object this is the URI object

The URI Object contains the following fields.

Field Type Description
cmd String The HTTP host variable.
uri String Uniform Resource Identifier, what people normally type in their browser after the site name.
status Integer Return status value
t String Type of content
mime String mime type

DNS

Field Type Description
id Integer The ID for the query provided by the client
query Object A query by a client

DNS query object

Field Type Description
flags Array Array of DNS Flags
questions Array Request by client for a lookup

Questions

Field Type Description
name String Name being requested. DNS names end in a period (.)
type String Record Type requested
class String DNS Class

Answers

Field Type Description
name String Name being requested. DNS names end in a period (.)
ttl Integer Time to Live. How many seconds this answer should be cached by the client.
type String Record Type requested
class String DNS Class
cname String Conical Name reference. When this is provided, the client should request an IP address from this system.
ipv4 String The returning address is an IPv4 address
ipv6 Sting The returning address is an IPv6 address

Event Values

The base flow record is:

Field Type Description
@message String The raw message unless it is in JSON messages, the event name.
@timestamp Integer GMT in Linux Epoch time
@level String System message level.
@source String Where the message was generated
@tags Array Array of information labels
@incidents Array Array of incident (issue) labels
@facility String Facility of the source
@sender String device that sent the alarm
@fields Object Parsed data of the message

AWS CloudTrail

Fluency downloads and populates the @field with the AWS CloudTrail JSON log. Learn more about CloudTrail here:

There are key fields to pay attention to:

Field Type Description
eventName String Action
userIdentity.username String Username of the action
sourceIPAddress String Source of the connection that made the API request

There is no defined list of eventNames.

Office 365

The values of the @fields attribute will contain the JSON record of the audit. Since this record is retrieved as a JSON object, the @message field will contain the @fields.Operation value.

An official list of the fields are here:
https://docs.microsoft.com/en-us/office365/securitycompliance/detailed-properties-in-the-office-365-audit-log

Descriptions are from the Office MS site when a definition existed.

Field Type Description
CreationTime String GMT String of the time
Id String The ID of the report entry. The ID uniquely identifies the report entry.
Operation String The name of the user or admin activity. The value of this property corresponds to the value that was selected in the Activities drop down list. If Show results for all activities was selected, the report will included entries for all user and admin activities for all services. For a description of the operations/activities that are logged in the Office 365 audit log, see the Audited activities tab in Search the audit log in the Office 365 Security & Compliance Center. For Exchange admin activity, this property identifies the name of the cmdlet that was run.
OrganizationId String The GUID for your Office 365 organization.
RecordType Integer The type of operation indicated by the record. The following values indicate the record type.
UserKey String An alternative ID for the user identified in the UserIDproperty. For example, this property is populated with the passport unique ID (PUID) for events performed by users in SharePoint. This property also might specify the same value as the UserID property for events occurring in other services and events performed by system accounts.
Version Integer Indicates the version number of the activity (identified by the Operation property) that’s logged.
Workload String The Office 365 service where the activity occurred. The possible values for this property are: SharePoint OneDrive Exchange AzureActiveDirectory DataCenterSecurity Compliance Sway SecurityComplianceCenter PowerBI MicrosoftTeams ThreatIntelligence
ClientIP String The IP address of the device that was used when the activity was logged. The IP address is displayed in either an IPv4 or IPv6 address format.
ClientIPAddress String SharePoints version of ClientIP
ObjectId String For Exchange admin audit logging, the name of the object that was modified by the cmdlet. For SharePoint activity, the full URL path name of the file or folder accessed by a user. For Azure AD activity, the name of the user account that was modified.
UserId String The user who performed the action (specified in the Operation property) that resulted in the record being logged. Note that records for activity performed by system accounts (such as SHAREPOINT\system or NT AUTHORITY\SYSTEM) are also included in the audit log.
UserDomain String Identity information about the tenant organization of the user (actor) who performed the action.
CorrelationId String Unlisted Attribute. The attribute appears to relate a request with its response. Appears in Sharepoint logs when a search is followed by an FileUploaded.
EventSource String Identifies that an event occurred in SharePoint. Possible values are SharePoint and ObjectModel. (Sharepoint)
ExternalAccess String For Exchange admin activity, specifies whether the cmdlet was run by a user in your organization, by Microsoft datacenter personnel or a datacenter service account, or by a delegated administrator. The value False indicates that the cmdlet was run by someone in your organization. The value True indicates that the cmdlet was run by datacenter personnel, a datacenter service account, or a delegated administrator. For Exchange mailbox activity, specifies whether a mailbox was accessed by a user outside your organization. (Exchange)
ItemType String The type of object that was accessed or modified. Possible values include File, Folder, Web, Site, Tenant, and DocumentLibrary. (SharePoint)
ListId String Unlisted Attribute
ListItemUniqueId String Unlisted Attribute
Site String The GUID of the site where the file or folder accessed by the user is located. (Sharepoint)
UserAgent String Information about the user’s browser. This information is provided by the browser. (Sharepoint)
WebId String Unlisted Attribute. Web Hash Identifier
SourceFileExtension String The MS DOS file extension used for application mapping
SiteUrl String The URL used to connect to this resource
SourceFileName String The filename by itself with extension
SourceRelativeUrl String The relative directory. This plus site, plus filename is the complete URL

Record Types

Value Meaning
1 Indicates a record from the Exchange admin audit log.
2 Indicates a record from the Exchange mailbox audit log for an operation performed on a singled mailbox item.
3 Also indicates a record from the Exchange mailbox audit log. This record type indicates the operation was performed on multiple items in the source mailbox (such as moving multiple items to the Deleted Items folder or permanently deleting multiple items).
4 Indicates a site admin operation in SharePoint, such as an administrator or user assigning permissions to a site.
6 Indicates a file or folder-related operation in SharePoint, such as a user viewing or modifying a file.
8 Indicates an admin operation performed in Azure Active Directory.
9 Indicates OrgId logon events in Azure Active Directory. This record type is being deprecated.
10 Indicates security cmdlet events that were performed by Microsoft personnel in the data center.
11 Indicates Data loss protection (DLP) events in SharePoint.
12 Indicates Sway events.
14 Indicates sharing events in SharePoint.
15 Indicates Secure Token Service (STS) logon events in Azure Active Directory.
18 Indicates Security & Compliance Center events.
20 Indicates Power BI events.
22 Indicates Yammer events.
24 Indicates eDiscovery events. This record type indicates activities that were performed by running content searches and managing eDiscovery cases in the Security & Compliance Center. For more information, see Search for eDiscovery activities in the Office 365 audit log. 25, 26, or 27 - Indicates Microsoft Teams events.
25, 25 or 27 Indicates Microsoft Teams events.