Azure Event Hubs
Integration with Microsoft Azure Event Hub
This guide outlines how to configure Azure Cloud to export Event Hubs to Fluency.
Create a Resource group
Open the Microsoft Azure portal (https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create), and navigate to the Resource groups page.
Click the "+ Create" button on the right side of the page to create a new Resource group.
Under Project details, select a Subscription, and give the Resource group a name.
Under Resource details, choose a Region. (Default: East US)
Click "Review + Create" to continue to the next page.
A Resource group is created as below.
Create an Event Hub Namespace
The Event Hub Namespace will contain one or more Event Hubs. The configured Azure services will create Event Hubs in this namespace to store activity logs and diagnostics logs.
Fill in the information needed (similar to last section) to create an Event Hub Namespace.
Wait for deployment to complete.
Complete.
Create an Event Hub
Navigate to the Event Hub (Namespace) resource created above:
Navigate to Event Hubs, under Entites. Click the "+ Event Hub" button at the top to create a new Event Hub:
Give the Event Hub a name, and choose "Create":
Create Event Hub Access Policy for Fluency
Select (click on) the newly created Event Hub from the list:
Under Settings, navigate to the Shared access policies:
Add a new Policy for the Fluency integration.
Select the policy to view the Key strings:
Make a note of the Keys shown, especially the "Connection string–primary key".
Notes on the Connection string–primary key
NOTE: One common issue is creating a 'Shared access policy' in the wrong place.
Verify that the "Connection string–primary key" ends with a string similar to ";EntityPath=\<eventhubname>".
On the Azure page where the token was obtained, the sub header should says "Event Hubs Instance". (Notice in the picture above, it's the line under "fluency-audit")
If the page instead say "Event Hubs Namespace", then it's the wrong page, and you will need to actually create/use the Event Hub instance inside the Namespace. The two pages are very similar otherwise.
Adding a Fluency plug-in for Azure Event Hub
Login to the Fluency Cloud portal: https://(companyname).cloud.fluencysecurity.com.
Open the Main Menu from the upper left-hand corner and choose the Cloud Integrations option under the Data Ingress section.
On the following page, navigate to the Cloud Infrastructure as a Service section.
To Add an integration for Azure Event Hubs, choose the "Azure EventHub" icon from the group on the left side of the page to create a new Event Hubs integration endpoint
NOTE: If an integration endpoint was setup previously, you can also select and modify it from the rigt side of the page.
In the pop-up window, choose the “Initialize” button to add the integration endpoint.
Select the Azure EventHub integration endpoint from the list on the right side of the page, in the “Cloud Infrastructure as a Service” section. Choose the pencil icon to edit/configure the connector.
On the following page, enter the "Connection string–primary key" from the previous section, and give the Event Hub integration a short Name/Description:
Click the "+ ADD EVENT HUB" button. (Multiple Event Hub connections can be added within the Azure integration.)
Click "SAVE" to commit the changes and finish adding the integration.
Note: At this point, the Event Hub may not have data.
The following section will detail the process to generate some data to your new Event Hub.
Appendix: Sending Azure Audit logs to Event Hub
Open the Microsoft Azure portal and select Azure Active Directory > Monitoring > Audit logs
On the Audit Logs page, select Export Data Settings:
Navigate to the Diagnostics settings pane, and choose "+Add diagnostics setting":
On the following page, Select the desired log categories and choose the "Stream to an event hub" option:
Configure the Event hub settings to match the Namespace and Event Hub created in the previous section. Save and wait for completion.
If no Event hub name is specified above, an Event hub is created in the namespace with the default name "insights-logs-audit".
After about 15 minutes, verify that events are displayed in your event hub. To do so, go to the event hub from the portal and verify that the "incoming messages" count is greater than zero.
Events from other sources within Azure can be configured in a similar manner. Refer to the References section for more information.
References
Follow the Microsoft guides below to configure sources to send to the new Event Hub.
Azure Monitor
https://docs.microsoft.com/en-us/azure/azure-monitor/overview
https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs
https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings?tabs=CMD
Monitoring data available
Sources of monitoring data for Azure Monitor describes the different tiers of data for Azure applications and the kinds of monitoring data available for each. The following table lists each of these tiers and a description of how that data can be streamed to an event hub. Follow the links provided for further detail.
| Monitoring data available | ||
|---|---|---|
| Tier | Data | Method |
| Azure tenant | Azure Active Directory audit logs | Configure a tenant diagnostic setting on your Azure Active Directory tenant. See Tutorial: Stream Azure Active Directory logs to an Azure event hub. |
| Azure subscription | Azure Activity Log | Create a log profile to export Activity Log events to Event Hubs. For more information, see Stream Azure platform logs to Azure Event Hubs. |
| Azure resources | Platform metrics Resource logs | Both types of data are sent to an event hub using a resource diagnostic setting. See Stream Azure resource logs to an event hub. |
| Operating system (guest) | Azure virtual machines | Install the Azure Diagnostics Extension on Windows and Linux virtual machines in Azure. See Streaming Azure Diagnostics data in the hot path by using Event Hubs for details on Windows VMs and Use Linux Diagnostic Extension to monitor metrics and logs for details on Linux VMs. |
| Application code | Application Insights | Use diagnostic settings to stream to event hubs. This tier is only available with workspace-based Application Insights resources. For help with setting up workspace-based Application Insights resources, see Workspace-based Application Insights resources and Migrate to workspace-based Application Insights resources. |
Additional links
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-about
Sample
A typical event obtained from the Azure Event Hub integration:
Page last updated: 2023 Aug 09