The Fluency Cloud architecture is structured using the AWS environment. Each client account has its own unique processing, storage, and database isolation. Inherent in our design is the ability to rapidly scale and drive capacity on-demand. Our Lava DB is designed using horizontal shards with dynamic elasticity that does not require re-provisioning. With AWS compute and storage elements those are also on-demand and will automatically provision what is needed and when it is needed.
Our approach differs from other vendors that have “ported” their solutions to the cloud. We built ours from the ground up to ensure high performance, capacity and scale-ability using proprietary techniques. This ensures we have an ongoing overall lower cost of goods than our competitors and therefore we can bring those savings to our clients.
Fluency Cloud is not the same architecture as the previous Fluency Security Analytics Server. Fluency Cloud addresses the limitations in an on-premise design while taking advantage of AWS cloud capabilities. Log correlation and vision remain the core of our product. Fluency Cloud further addresses the needs of:
Unlike all other data-centric offerings, Fluency Cloud is built directly on the AWS infrastructure taking full advantage of the AWS architecture. This differs from the major vendors that have ported their appliance-based software to the Cloud in a virtual image. This low-end technique reduces performance and increases cost. By applying an AWS native solution, Fluency can transition cost savings into user capability, such as offering the industry’s only one-year cold storage by default.
Fluency Cloud is implemented in AWS Virtual Private Cloud (VPC). All customers' computation resources are fully isolated from each other. Each instance only processes data for one customer. Customers receive dedicated/isolated resources.
Fluency Cloud provides one outwardly accessed point of communication on SSL:
Web Interface: Person
API Interface: Fluency Agent
Connection to the system is by its system name, which is a corporate tag followed by ‘cloud.fluencysecurity.com.’ This system name is registered in the domain name system (DNS). When referring to the connection point, regardless of web or syslog, the DNS name should be used.
Where is the management port? Management is done via a connection from the AWS host system to a secure shell (ssh) on the guest server. This connection is accomplished via a key-based authenticated ssh connection to the host, and then a second key-based connection to the guest server. These connections are protected by multiple level ACLs, instance level security group, and subnet level ACL. The combination of the web, ssl-agent, and ssh services is the total perimeter of the server and marks the common criteria target of evaluation of a Cloud server.
Fluency collectors that leverage the Fluency agent are preferred, but not required. Log Fluency collectors allow for store and forward process, which can store streamed system log (UDP) during network connectivity issues. The Fluency agent generates an authenticated SSL tunnel to the Fluency Cloud instance.
In the case where there is no Fluency device, a VPN tunnel can be established from the customer's premise to the Fluency Cloud. In this case, syslog traffic is sent through the tunnel to a price address inside the tunnel. In production, syslog is not to be sent in the clear to the private Cloud.
The amount of data being sent to Fluency is minimal. Fluency represents only a very small percentile of the upload data capacity. Modern communication lines are bidirectional, meaning sending data does not interfere with receiving data. It also means to most organizations, the amount of download data is significant, while an upload is a fraction of the download used.
While the initial username and passphrase are single-factor authentication, Fluency Cloud implements OAuth authentication, which can be set to two-factor authentication. This is the preferred mechanism for authentication. Both Google and Azure are supported.
To further protect the interface, the system’s internal firewall can be set to accept communication only from the source range of your organization.
Fluency Cloud is not a single virtual instance running in the cloud, but a native AWS application that takes advantage of the unique storage, processing and network configurations available for Cloud applications. Fluency Cloud is the first central log management system designed specifically for the Cloud.
This is a single public subnet for each customer. This node acquires the public network address used for Internet connections. An optional VPN server for direct syslog communication could be placed in this subnet. The main web application server runs in this subnet.
For each public instance, there is a private subnet. For the private subnet, there is no public IP address. All Internet traffic is required to go through a virtual private Cloud (VPC) network address translation (NAT) gateway. All internal processing, such as the data store, event fusion module and analytics engine, run on this private subnet.
There is a third subnet that is defined as AWS public but has no Internet-facing service. This subnet is for on-demand searches. All nodes are dynamic in this subnet. On-demand searching requires a public address to bypass the bandwidth limitation of an AWS NAT gateway.
On AWS, every subnet is attached to a network access control list (ACL). This is similar to the internal Linux firewall, iptables. Every instance is attached to a security group, in which access control lists are defined. This is a "share nothing" structure. The only shared resources are load balancer and NAT gateway.
Fluency's AWS implementation has a five nine (99.999) availability and thirteen nines (99.99999999999) or durability. There are a number of other enhancements by Fluency that increase the data integrity, including:
Store and Forward
Local collectors act in store & forward mode. This means data is stored in the collector until validation of it being received. It is more likely there a line failure, than a drive failure. Store & forward allows Fluency to address this type of error.
Fluency also is a native AWS design. This means processing and data storage are assigned dynamically. This allows Fluency to acquire more processing power if there is a need, or a failure. It also means Fluency spins up more disk to address capacity and performance. In this manner, the common issues of data integrity, such as processor overload and not enough disk space, is addressed.
Significant engineering and thought has been put into implementing the Fluency Cloud solution. Its design is not simply to be placed in the Cloud for access, but to take advantage of the AWS architecture to address operational needs. Cloud architectures have significant advantages over on-premise, such as dynamic allocation, disk speed, and data retention.