API Polling
Most of the template catalog collects this way. Fluency calls the vendor's API from its own infrastructure, on a schedule, using a credential you create in the vendor's console and paste into the template's install form. Nothing is installed on your network and no inbound port is opened. Each call asks for everything since the last one, writes what comes back, and remembers where it got to.
This page covers what every API integration has in common. The per-vendor guides in Integrations cover what is particular to one product — which permissions to grant, which host to use, which streams arrive — and link back here for the rest.
How It Works
- You create a credential in the vendor's console — an API token, an OAuth client, a key pair — and give it read access to the logs.
- You install the vendor's template from Platform → Applications → Install Application From Template and paste the credential into its parameters.
- The installed application polls. On each cycle it authenticates, requests records newer than its last position, hands them to the template's parser, writes them to the data lake index named on the form, and moves its position forward.
In the application's pipeline diagram the collector appears as a Plugin Data Source node feeding the template's router. The same node type appears on Platform → Configuration for every API-polled source.
All traffic is outbound from Fluency to the vendor over HTTPS. The calls come from Fluency's cloud infrastructure, not from your network — which matters for one setting that appears in many vendor consoles, covered under IP restrictions.
The one exception is a directory or service that only exists inside your network. Fluency cannot reach it from the cloud, so the poll runs from a Fluency Collector on that network instead — see LDAP.
The Credential
Credential Shapes
Vendors issue read credentials in a handful of shapes, and the shape decides more about the integration's lifetime than anything on the Fluency side does.
| Shape | What you paste | Examples | The thing to know |
|---|---|---|---|
| API token tied to a user | A token string, usually with the vendor's host | Okta, SentinelOne, Varonis, Qualys | The token carries its creator's permissions, live. Edit that person's role and the token's reach changes with it; deactivate the person and the integration stops. Issue it from a service account. |
| API token with its own role | A token string and an endpoint or account ID | Trend Vision One, Cloudflare, Abnormal Security | Choose the role on the token itself. Most of these have a fixed lifetime. |
| Signed-request key pair | An integration or key ID plus a secret, and the API host | Cisco Duo, Cortex XDR, Mimecast (Legacy) | The secret never travels; each request is signed with it. Where the vendor offers more than one signing mode, the template has a parameter that has to match. |
| Principal and secret | A service principal and its secret, plus the API host | Proofpoint TAP, Proofpoint Essentials | Sent as HTTP Basic credentials on every request. |
| OAuth client credentials | A client ID and client secret, sometimes a tenant or base URL | CrowdStrike, Sophos, Microsoft Defender, Office 365 Audit, Bitwarden, Black Kite, box.com, Salesforce, Mimecast CG | Fluency exchanges the pair for a short-lived access token on each cycle. The scopes are set on the client, and vendors do not add new scopes to existing clients retroactively. |
| Refresh token | Client ID, secret and a long-lived refresh token | Workday | The refresh token is the long-lived part; whether it expires is a checkbox in the vendor's console. |
| Service-account key | A JSON key file's contents and the administrator to act as | Google Workspace (Service Account) | The key never expires, which makes it a review item rather than a rotation item. |
| Consent grant | An administrator's email address — no secret at all | Office 365 Activity API, Office 365 Resource Watch, Azure AD Audit, Google Workspace (OAuth) | The install sends that administrator a request to approve Fluency's application in their tenant. Until they approve it, the application runs and collects nothing. |
Least Privilege, and Where It Bites
Grant read access to the logs and nothing else — every guide names the exact permissions. Three patterns recur where the obvious least-privilege answer is the wrong one:
- A second permission buys a second feed. Templates that also sync users or devices onto SIEM → Resources usually need a separate permission for that half. Leave it off and the event feed works while the roster stays empty, with no error anywhere. Cisco Duo is the clearest case.
- The permission filters rather than fails. A few vendors apply the reading account's own visibility to the audit log, so an under-permissioned credential gets fewer rows rather than an error. Workday works this way.
- The vendor pre-selects the most powerful role. Some credential wizards open on full read/write. Change it before saving — Black Kite opens on Super Admin.
IP Restrictions on the Credential
Many vendor consoles let you pin a credential to a list of source addresses. Leave this off. Fluency polls from managed cloud infrastructure whose egress addresses you do not control and are not published as stable, so a credential pinned to your office ranges is rejected on every call. The failure is easy to misread, because the credential itself is valid: Okta answers with an authentication error, CrowdStrike with 403 access denied, authorization failed, and re-issuing the secret changes nothing.
If your policy requires an address restriction, get the egress range from Fluency support first and treat any change to it as a maintenance event.
Host and Region
Most vendors partition their API by region or tenant, and the template asks for a base URL, a regional host, or a tenant name. A wrong value rarely says so. Across the guides, a wrong host most often produces exactly the error a bad credential produces — Okta's 401 E0000011, Duo's 401 40102, CrowdStrike's Failed to generate access token, Trend Vision One's 401 — so a reader re-issues a perfectly good secret several times before checking the host. Some vendors are stranger still: Workday answers an unknown tenant with 503 Service unavailable, which reads as a temporary outage, and a Cloudflare zone ID in place of the account ID installs cleanly and collects nothing.
When an API integration fails authentication on a credential you have just created, check the host first.
Installing
Install from Platform → Applications → Install Application From Template; the steps are on Integrations → Connecting a Source. Two things apply to every API template:
- Parameters are read-only once installed. Changing a credential, a host or an index means uninstalling and installing again — see Rotating the Credential.
- The data lake index name has to be unique within the lake. Two applications writing the same index name in the same lake collide, and the second one aborts with the reason in its Error Message.
What Arrives, and When
The First Poll
A new install does not start empty and does not reach back forever. Its first poll backfills a window that depends on the template and the vendor's API, and after that it collects incrementally.
| Template | First poll reaches back | Later polls |
|---|---|---|
| CrowdStrike Falcon | 72 hours of alerts | Alerts every 5 minutes; host roster every 12 hours; event stream held open |
| Microsoft Defender | 24 hours | Every 5 minutes |
| Okta | 7 days | Incremental |
| box.com | Up to the two-week stream | Every 300 seconds |
| Bitwarden | 30 days | Incremental |
Some sources also have a delay of their own between an event and its availability over the API: Office 365 can take up to 12 hours to deliver the first content on a new subscription, Salesforce's hourly event files lag three to six hours, and Google Workspace's token activity lags by a couple of hours. An empty index in the first hours is not, on its own, a fault.
The cadence a given install actually runs at is recorded on its data source, and many of the per-vendor guides give it too.
The API Window, and Why Outages Can Be Permanent
A vendor's API often serves a much shorter history than its console keeps. Collection that stops for longer than the API window cannot be backfilled when it restarts: the records are still in the vendor's console but no longer reachable through its API.
| Source | API window | Console keeps |
|---|---|---|
| Sophos Central (SIEM API) | 24 hours | 90 days |
| Proofpoint Essentials | 3 days | — |
| Proofpoint TAP, Mimecast CG | 7 days | — |
| Office 365 | 7 days per content blob | — |
| Entra ID audit | 7 days (Free) / 30 days (P1, P2) | — |
| box.com (streaming feed) | 2 weeks | 1 year |
| Okta | 90 days | 90 days |
| Cisco Duo | 180 days, less if the tenant purges sooner | — |
The shorter the window, the less time you have to notice a stopped integration. For the short-window sources, a regular check that collection is still advancing — the vendor-side signals under Confirming It Works — is worth more than any other monitoring.
Duplicates and Ordering
Polling feeds are not exactly-once. A reinstall starts a new application with no memory of the old one's position, so its first poll backfills ground already collected; some vendors also deliver duplicates deliberately so that nothing is lost. Office 365 says outright that removing them is the SIEM's job. Where it matters, deduplicate on the vendor's own event ID and sort on the vendor's timestamp rather than on arrival order:
| Source | Deduplicate on |
|---|---|
| Okta | uuid (sort on published) |
| box.com | event_id |
| Office 365 | Id |
| Google Workspace | uniqueQualifier with time |
| Cisco Duo | txid |
A few feeds return state rather than events — the same alert or finding coming back each time its status changes. CrowdStrike alerts, Black Kite findings and Varonis alerts all behave this way; the guides name the key to deduplicate on.
Events and Resources
Several templates collect two things: an event feed into a data lake index, and a roster of users, devices or groups onto SIEM → Resources. The roster is what turns an opaque ID in an event into a name, and because a populated tenant always has users or hosts, it is also the stricter health check — an event feed on a quiet weekend can be legitimately empty, a roster cannot. On the Resources page, Total Records 0 / No data found means the sync ran and found nothing; Resource Load Failed means nothing has been collected for that record set.
Confirming It Works
A Running badge means the pipeline started. It does not mean the vendor accepted the credential, and on a consent template it means waiting for approval. Confirm with something the vendor's side records:
- The vendor's own record of the credential being used. Most consoles show a last-used time or an API audit trail — Okta's token status, Proofpoint's Last Used, Cloudflare's Last used, CrowdStrike's API audit trail. A credential that has never been used has never been tried.
- A known event. Do something you can find again — a sign-in, a test detection such as EICAR, a file share — and search for it after a poll cycle has had time to run.
- The roster, where there is one. It should have rows.
- The application itself. Platform → Applications for the badge and any red !, Platform → Usage for ingress by index, Data Lake → Management for the index's Last Imported time. Confirm Data Is Arriving walks through all three.
Rotating the Credential
Because parameters are read-only, every rotation is an uninstall and a reinstall with the new value. Reinstall with the same datalake index name so the data continues in one place — which is also why the old application has to go first, since two applications cannot write the same index. How much the rotation costs depends on whether the vendor lets two credentials be valid at once.
- The vendor allows two — rotate with an overlap. Create the new credential and leave the old one valid. Uninstall and reinstall with the new one, confirm collection has resumed, and only then revoke the old one. The only gap is the minute the reinstall takes, and the next poll covers it. Entra ID client secrets, Google service-account keys, Trend Vision One and Cortex XDR keys, and Sophos and Varonis credentials all work this way.
- The vendor keeps one — rotate as a cut-over. Some vendors hold exactly one secret per client and invalidate the old one the moment a new one is issued: Duo, Bitwarden, box.com, CrowdStrike, Cloudflare's Roll, Workday, Mimecast. Reinstall immediately after regenerating. Collection is down for as long as that takes, and whether the gap is recovered depends on the API window.
Either way, the new install starts without the old one's position, so expect its first poll to overlap what was already collected.
Record the expiry date when you create the credential, not when it fails:
| Credential | Lifetime |
|---|---|
| Entra ID client secret (Microsoft templates) | 180 days by default, 24 months at most |
| SentinelOne user API token | 180 days |
| Trend Vision One API key | 1 year by default |
| Black Kite client | About 2 years |
| Sophos Central credential | 36 months; cannot be renewed |
| Okta API token | Revoked after 30 days unused |
| Google OAuth refresh token | Revoked after 6 months unused |
| Workday refresh token | Non-expiring only if that option was ticked |
| Google service-account key, Microsoft admin consent | Do not expire — review periodically instead |
When It Stops
| Symptom | Usual causes |
|---|---|
| Aborted right after install | The index name is already used in that lake, or a required parameter is malformed. The details panel's Error Message says which. |
| Running, nothing has ever arrived | Wrong host or region (reads as a bad credential); the secret's ID pasted instead of its value; an IP restriction on the credential; consent not yet approved; a licence the stream requires. Or simply inside the first-content delay above. |
| Worked for weeks, then stopped | The credential expired; its owning user was deactivated or had their role changed; consent was revoked by another administrator; the token was revoked for being idle. |
| Some streams arrive, others never do | A permission missing for one API while the others are granted — that endpoint answers 403 while the rest keep working. Or a scope added to the vendor's API after the credential was created, which existing clients do not get automatically. |
| Arrives, but fewer records than the vendor's console shows | The credential's own visibility filters the log; the API window is shorter than the console's; or the stream is a roll-up of what the console shows individually. |
| Gaps under load | The vendor's rate limit, sometimes shared with every other consumer of the same tenant — Okta gives each token a share of the organisation's bucket. |
| Duplicates after a reinstall | Expected; see Duplicates and Ordering. |
Sources With No Template
A few products are connected through a connector on Platform → Integrations, with the data source, router and sink then added by hand on Platform → Configuration. That path still works but is no longer the normal one — new sources are delivered as templates. The pages for those products say so at the top; the Integration Matrix lists them.
Related
- Integrations — the template catalog and every per-vendor guide.
- Connecting to AWS — the role-based credential the AWS templates use, which works differently from everything on this page.
- Install Application From Template — every template's parameters.
- No Data Arriving — the layer-by-layer diagnosis.