Architecture
This page contains information to aid in the understanding of how the internals of l2met work.
L2met receives collections of logs via an HTTP server, extracts measurements from the logs into buckets. Periodically, aggregated buckets are flushed to a store. The store is in-memory in a single process formation or Redis in a multi process formation. Outlets scan buckets from the store, compute statistics on the buckets and deliver the resulting metrics to the respective outlet. (e.g. Librato)
All of the data models and processes are documented in the l2met code base and are browsable via godoc.
We will consider two paths of data flow. Incoming data represents the logs, extracting data from logs, storing data as buckets in a memory store. Outgoing data represents reading buckets from the memory store, computing statistics on the buckets and out-letting the metrics to services like Librato.
The HTTP request is authenticated, authorized, and parsed and its body is placed into a buffer. The response is returned to the request and the un-parsed body is placed into the Inbox for further processing.
The un-parsed HTTP request body is parsed into Bucket structs and pointers to the structs are buffered in the register for a short period of time. When adding a bucket to the register, we check to see if a bucket's ID is already in the register, if it is we merge the values of the buckets otherwise we add the bucket to the register. This allows us to buffer measurements before flushing the buckets to the Store.
On a flush interval, the transferee places the buffered buckets into the Outbox so that the outlet routine can send the buckets to the store. The store can be in-memory or a Redis server.
Multiple outlet processes can run simultaneously. Buckets can be partitioned by the hash of their Id. We use Redis as a mutex to coordinate which outlet process can access a particular partition. This ensures that we can properly compute statistics on a bucket.
When the receiver's outlet sends data to Redis, it encodes the bucket's Id using a binary format. The outlet uses the encoded Id as the key to a Redis list. We RPUSH the values of the bucket onto the list.
When the receiver's outlet pushes a bucket onto a list, it also adds the Id of the bucket to a set. They key to the set is composed of the hashed Id of the bucket and the time at which the bucket will be ready for processing. E.g. If a bucket has a resolution of 60s and the bucket is added to Redis at t=0, the bucket will not be complete for another 59s. Hence, the key of the set will be t59 + hash(bucket.Id) and the value added to the set will be the encoded Id of the bucket.
The reader is scheduled to look for buckets to process each second. It acquires a lock for a partition, reads bucket ids from the bucket partition set, and fetches the bucket data from the bucket list. It places a filled in Bucket struct into the outlet's Inbox.
The converter reads the buckets from the Inbox, computes the statistical functions on the bucket data, then adds them to a map which is keyed by the bucket's encrypted Librato API tokens. The outlet must submit HTTP requests to librato with user supplied Librato credentials which is the motivation for grouping the bucket dat a.
The outlet takes the prepared metrics from the Outbox, converts the data into JSON and makes an HTTP request to the Librato API.