Normalization

Table of contents

Base Record
Flow Values
1. DHCP Data
2. HTTP
3. DNS
Event Values
1. AWS CloudTrail
2. Office 365

Data Normalization is mapping the attribute-value pairings into a consistent naming and type convention.

Base Record

Field	Type	Description
@type	String	The type of record this is: metadata or event

Flow Values

The base flow record is:

Field	Type	Description
start_ms	Integer	GMT in Linux Epoch time
dur	Integer	duration in milliseconds of the session length
proto	Integer	Protocol Number
sip	String	source address, often the source IP address
sp	Integer	Source port
dip	String	destination address, often the destinations IP address
dp	Integer	Destination port
rxP	Integer	Received number of packets
txP	Integer	Transmitted number of packets
rxB	Integer	Received number of bytes
txB	Integer	Transmitted number of bytes
rf	Integer	Mask of the combined received flags
totalB	Integer	Total of Bytes in session
partition	String	name of shard
dHost	String	Passive mapped name from DNS query
http	Object	HTTP protocol metadata
dns	Object	DNS protocol metadata
meta	Object	Device related metadata.
c	String	Collector who produced the message

The base object often has a geo-ip lookup.

Field	Type	Description
s_country	String	Two letter Internet country code of source
s_city	String	City related to destination by IP address of source
s_org	String	Registered Organization owner of source
s_isp	String	Registered ISP of source
d_country	String	Two letter Internet country code of destination
d_city	String	City related to destination by IP address of destination
d_org	String	Registered Organization owner of destination
d_isp	String	Registered ISP of destination

The base object might have an associated user from LDAP logs.

Field	Type	Description
su	String	The domain name of the source’s user
du	String	The domain name of the destination’s user

DHCP Data

Field	Type	Description
hostname	String	Referenced to Hostname
mac	String	Network Interface Card Machine Address (data link level)

HTTP

Nested in the HTTP root attribute are the attributes for an HTTP connection.

Field	Type	Description
host	String	The HTTP host variable.
agent	String	the type of process that is running this request
referer	String	The URI that caused this page to be requested
xforward	String	If a firewall has X-Forward on, this field will show the address of the internal system. Fluency will generate a second record to correlate this activity from source to destination.
uris	Object	this is the URI object

The URI Object contains the following fields.

Field	Type	Description
cmd	String	The HTTP host variable.
uri	String	Uniform Resource Identifier, what people normally type in their browser after the site name.
status	Integer	Return status value
t	String	Type of content
mime	String	mime type

DNS

Field	Type	Description
id	Integer	The ID for the query provided by the client
query	Object	A query by a client

DNS query object

Field	Type	Description
flags	Array	Array of DNS Flags
questions	Array	Request by client for a lookup

Questions

Field	Type	Description
name	String	Name being requested. DNS names end in a period (.)
type	String	Record Type requested
class	String	DNS Class

Answers

Field	Type	Description
name	String	Name being requested. DNS names end in a period (.)
ttl	Integer	Time to Live. How many seconds this answer should be cached by the client.
type	String	Record Type requested
class	String	DNS Class
cname	String	Conical Name reference. When this is provided, the client should request an IP address from this system.
ipv4	String	The returning address is an IPv4 address
ipv6	Sting	The returning address is an IPv6 address

Event Values

The base flow record is:

Field	Type	Description
@message	String	The raw message unless it is in JSON messages, the event name.
@timestamp	Integer	GMT in Linux Epoch time
@level	String	System message level.
@source	String	Where the message was generated
@tags	Array	Array of information labels
@incidents	Array	Array of incident (issue) labels
@facility	String	Facility of the source
@sender	String	device that sent the alarm
@fields	Object	Parsed data of the message

AWS CloudTrail

Fluency downloads and populates the @field with the AWS CloudTrail JSON log. Learn more about CloudTrail here:

There are key fields to pay attention to:

Field	Type	Description
eventName	String	Action
userIdentity.username	String	Username of the action
sourceIPAddress	String	Source of the connection that made the API request

There is no defined list of eventNames.

Office 365

The values of the @fields attribute will contain the JSON record of the audit. Since this record is retrieved as a JSON object, the @message field will contain the @fields.Operation value.

An official list of the fields are here:
https://docs.microsoft.com/en-us/office365/securitycompliance/detailed-properties-in-the-office-365-audit-log

Descriptions are from the Office MS site when a definition existed.

Field	Type	Description
CreationTime	String	GMT String of the time
Id	String	The ID of the report entry. The ID uniquely identifies the report entry.
Operation	String	The name of the user or admin activity. The value of this property corresponds to the value that was selected in the Activities drop down list. If Show results for all activities was selected, the report will included entries for all user and admin activities for all services. For a description of the operations/activities that are logged in the Office 365 audit log, see the Audited activities tab in Search the audit log in the Office 365 Security & Compliance Center. For Exchange admin activity, this property identifies the name of the cmdlet that was run.
OrganizationId	String	The GUID for your Office 365 organization.
RecordType	Integer	The type of operation indicated by the record. The following values indicate the record type.
UserKey	String	An alternative ID for the user identified in the UserIDproperty. For example, this property is populated with the passport unique ID (PUID) for events performed by users in SharePoint. This property also might specify the same value as the UserID property for events occurring in other services and events performed by system accounts.
Version	Integer	Indicates the version number of the activity (identified by the Operation property) that’s logged.
Workload	String	The Office 365 service where the activity occurred. The possible values for this property are: SharePoint OneDrive Exchange AzureActiveDirectory DataCenterSecurity Compliance Sway SecurityComplianceCenter PowerBI MicrosoftTeams ThreatIntelligence
ClientIP	String	The IP address of the device that was used when the activity was logged. The IP address is displayed in either an IPv4 or IPv6 address format.
ClientIPAddress	String	SharePoints version of ClientIP
ObjectId	String	For Exchange admin audit logging, the name of the object that was modified by the cmdlet. For SharePoint activity, the full URL path name of the file or folder accessed by a user. For Azure AD activity, the name of the user account that was modified.
UserId	String	The user who performed the action (specified in the Operation property) that resulted in the record being logged. Note that records for activity performed by system accounts (such as SHAREPOINT\system or NT AUTHORITY\SYSTEM) are also included in the audit log.
UserDomain	String	Identity information about the tenant organization of the user (actor) who performed the action.
CorrelationId	String	Unlisted Attribute. The attribute appears to relate a request with its response. Appears in Sharepoint logs when a search is followed by an FileUploaded.
EventSource	String	Identifies that an event occurred in SharePoint. Possible values are SharePoint and ObjectModel. (Sharepoint)
ExternalAccess	String	For Exchange admin activity, specifies whether the cmdlet was run by a user in your organization, by Microsoft datacenter personnel or a datacenter service account, or by a delegated administrator. The value False indicates that the cmdlet was run by someone in your organization. The value True indicates that the cmdlet was run by datacenter personnel, a datacenter service account, or a delegated administrator. For Exchange mailbox activity, specifies whether a mailbox was accessed by a user outside your organization. (Exchange)
ItemType	String	The type of object that was accessed or modified. Possible values include File, Folder, Web, Site, Tenant, and DocumentLibrary. (SharePoint)
ListId	String	Unlisted Attribute
ListItemUniqueId	String	Unlisted Attribute
Site	String	The GUID of the site where the file or folder accessed by the user is located. (Sharepoint)
UserAgent	String	Information about the user’s browser. This information is provided by the browser. (Sharepoint)
WebId	String	Unlisted Attribute. Web Hash Identifier
SourceFileExtension	String	The MS DOS file extension used for application mapping
SiteUrl	String	The URL used to connect to this resource
SourceFileName	String	The filename by itself with extension
SourceRelativeUrl	String	The relative directory. This plus site, plus filename is the complete URL

Record Types

Value	Meaning
1	Indicates a record from the Exchange admin audit log.
2	Indicates a record from the Exchange mailbox audit log for an operation performed on a singled mailbox item.
3	Also indicates a record from the Exchange mailbox audit log. This record type indicates the operation was performed on multiple items in the source mailbox (such as moving multiple items to the Deleted Items folder or permanently deleting multiple items).
4	Indicates a site admin operation in SharePoint, such as an administrator or user assigning permissions to a site.
6	Indicates a file or folder-related operation in SharePoint, such as a user viewing or modifying a file.
8	Indicates an admin operation performed in Azure Active Directory.
9	Indicates OrgId logon events in Azure Active Directory. This record type is being deprecated.
10	Indicates security cmdlet events that were performed by Microsoft personnel in the data center.
11	Indicates Data loss protection (DLP) events in SharePoint.
12	Indicates Sway events.
14	Indicates sharing events in SharePoint.
15	Indicates Secure Token Service (STS) logon events in Azure Active Directory.
18	Indicates Security & Compliance Center events.
20	Indicates Power BI events.
22	Indicates Yammer events.
24	Indicates eDiscovery events. This record type indicates activities that were performed by running content searches and managing eDiscovery cases in the Security & Compliance Center. For more information, see Search for eDiscovery activities in the Office 365 audit log. 25, 26, or 27 - Indicates Microsoft Teams events.
25, 25 or 27	Indicates Microsoft Teams events.