Option to cache Prometheus response #219

jesperpedersen · 2022-04-11T17:57:23Z

Have an option that will cache a Prometheus response for X seconds

The text was updated successfully, but these errors were encountered:

fluca1978 · 2022-05-26T14:03:37Z

Just curious: do you mean avoid to send out the result of metrics_page() at every single request but to serve it as it was x seconds before?
Is such functon so heavy to serve that requires caching?
Could it be a static temporary file the solution? I mean write out metrics_page() to the file and serve such a file is still valid.

jesperpedersen · 2022-05-26T14:16:05Z

Prometheus requests are light-weight for pgagroal (and pgmoneta, but not for pgexporter).

I would just cache it as a static string, together with the last served timestamp.

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. A new setting, named `metrics_cache` is defined and parsed as an "age" string. The value will be stored as a number of seconds. This required the refactoring of some configuration code, with particular regard to `as_logging_rotation_age()` in order to extract the "as age" logic into an utility function that can be used for different "age"-like strings. Moreover, some methods and configuration parameters have been changed from `int` to `unsigned int` (e.g., it does not make any sense to have a log rotation age negative!). The configuration related code has been changed to intercept wrong strings, e.g., when `log_rotation_age = -2m` the system disables the rotation and warns the user as the parameter is unknown. Also `+1m` is considered a bogus string. The `struct prometheus` has been expanded with an inner struct, named `struct prometheus_cache` that holds the cache payload and the timestamp the cache will be valid until. The `metrics_page()` has been refactored to check if the cache is configured and valid, in such case the response is served out of cache. On the other hand, the cache is reset and the response is served as a new one, with all the inner methods adding the result to the cache for the next time. A few utility functions have been introduced: - `is_metrics_cache_configured()` checks that the current configuration allows for the usage of cache, i.e., `metrics_cache` have been set; - `is_metrics_cache_valid()` checks hat the cache does handle some good stuff to be served; - `metrics_cache_reset()` to reset the cache, once it has been set within the shared memory; - `metrics_cache_add()` appends data to the cache. This methods is invoked whenever a new Prometheus message is spurt. In the case the overall amount of data overflows, the cache is made not valid. It is safe to call this function everywhere because it does nothing if the caching system is not enabled. If the cache overflows, it is set to "invalid" so that is not gong to be served anymore. The max size of the cache is defined as a constant value within the source code, and is set to 8kB (that should be reasonable since the average Prometheus response on a unloaded server is around 4kB). The documentation has been updated to reflect changes. Close agroal#239

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. A new setting, named `metrics_cache` is defined and parsed as an "age" string. The value will be stored as a number of seconds. This required the refactoring of some configuration code, with particular regard to `as_logging_rotation_age()` in order to extract the "as age" logic into an utility function that can be used for different "age"-like strings. Moreover, some methods and configuration parameters have been changed from `int` to `unsigned int` (e.g., it does not make any sense to have a log rotation age negative!). The configuration related code has been changed to intercept wrong strings, e.g., when `log_rotation_age = -2m` the system disables the rotation and warns the user as the parameter is unknown. Also `+1m` is considered a bogus string. The `struct prometheus` has been expanded with an inner struct, named `struct prometheus_cache` that holds the cache payload and the timestamp the cache will be valid until. The `metrics_page()` has been refactored to check if the cache is configured and valid, in such case the response is served out of cache. On the other hand, the cache is reset and the response is served as a new one, with all the inner methods adding the result to the cache for the next time. A few utility functions have been introduced: - `is_metrics_cache_configured()` checks that the current configuration allows for the usage of cache, i.e., `metrics_cache` have been set; - `is_metrics_cache_valid()` checks hat the cache does handle some good stuff to be served; - `metrics_cache_reset()` to reset the cache, once it has been set within the shared memory; - `metrics_cache_add()` appends data to the cache. This methods is invoked whenever a new Prometheus message is spurt. In the case the overall amount of data overflows, the cache is made not valid. It is safe to call this function everywhere because it does nothing if the caching system is not enabled. If the cache overflows, it is set to "invalid" so that is not gong to be served anymore. The max size of the cache is defined as a constant value within the source code, and is set to 8kB (that should be reasonable since the average Prometheus response on a unloaded server is around 4kB). The documentation has been updated to reflect changes. Close agroal#219

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. Two new configuration settings have been introduced: - `metrics_cache_max_age`, expressed in seconds, defines for how long a cached response can be served without having to compute a new one set of metrics. This parameter is required for the caching machinery to be enabled. - `metrics_cache_max_size` allows the user to avoid caching of responses greater than the expressed value in bytes. In any case, a cache response is tied to 8192 bytes, that sound resanobly for a Prometheus response since on my machine, with a couple of servers and an unload status, the response is around 4kB. If the cache overflows the 8192 memory limit, it is immediatly marked as invalid, that is the system is not going to serve the cached result. However, as soon as a new Prometheus request arrives, the cache will be re-filled from scratch, so if the current answer is lower than 8192 and lower than the optional `metrics_cache_max_size`, the response will be cached. Whenever an overflow of the cache happens, the user is warned. Due to the nature the cache is populated, the user could see more than one single advice in logs (at level DEBUG). The `struct prometheus` has been expanded with an inner struct, named `struct prometheus_cache` that holds the cache payload and the timestamp the cache will be valid until. The `metrics_page()` has been refactored to check if the cache is configured and valid, in such case the response is served out of cache. On the other hand, the cache is reset and the response is served as a new one, with all the inner methods adding the result to the cache for the next time. A few utility functions have been introduced: - `is_metrics_cache_configured()` checks that the current configuration allows for the usage of cache, i.e., `metrics_cache` have been set; - `is_metrics_cache_valid()` checks hat the cache does handle some good stuff to be served; - `metrics_cache_reset()` to reset the cache, once it has been set within the shared memory; - `metrics_cache_add()` appends data to the cache. This methods is invoked whenever a new Prometheus message is spurt. In the case the overall amount of data overflows, the cache is made not valid. It is safe to call this function everywhere because it does nothing if the caching system is not enabled. The documentation has been updated to reflect changes. Close agroal#219

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. Two new configuration settings have been introduced: - `metrics_cache_max_age`, expressed in seconds, defines for how long a cached response can be served without having to compute a new one set of metrics. This parameter is required for the caching machinery to be enabled. - `metrics_cache_max_size` allows the user to avoid caching of responses greater than the expressed value in bytes. In any case, a cache response is tied to 1MegaByte, by a value set in the source code. If the cache overflows the 1MB memory limit, it is immediatly marked as invalid, that is the system is not going to serve the cached result. However, as soon as a new Prometheus request arrives, the cache will be re-filled from scratch, so if the current answer is lower than 1MB and lower than the optional `metrics_cache_max_size`, the response will be cached. Whenever an overflow of the cache happens, the user is warned. Due to the nature the cache is populated, the user could see more than one single advice in logs (at level DEBUG). The `struct prometheus` has been expanded with an inner struct, named `struct prometheus_cache` that holds the cache payload, a lock (for concurrency), the size of payload, and the timestamp the cache will be valid until. New utility functions have been added: - `metrics_cache_append(char* data)` that appends the data to the cache in the case it is safe to do it, that is no overflow in the cache size. In case the data to append makes the cache to overflow, the cache is invalidated; - `metrics_cache_finalize(void)` set the timestamp for the cache vlidation. - `metrics_cache_invalidate(void)` set the cache as invalid by zero-filling the data. Used for example when an overflow is detected. - `metrics_cache_size_to_alloc(void)` computes the size, in bytes, required for the cache payload allocation. The size is set to the default, or `metrics_cache_max_size` if configured and in any case nothing more than the max size of 1MB. - `metrics_cache_alloc(struct prometheus* prometheus)` allocates the memory for the cache and sets the initial values. This has to be called when the `struct prometheus` is allocated and initialized. The `metrics_page()` method has been refactored to check if the cache is valid, in such case the response is served directly out of cache. If the cache is not valid, the function gets the cache lock and invokes several times the append function (in inner method calls). Last, it finalizes the cache and release the lock. If the optional configuration parameter `metrics_cache_max_size` changes, the system does not reallocate memory, rather it issue a warning about the need for a restart. The documentation has been updated to reflect changes. Close agroal#219 Modified the prometheus_cache structure to handle dynamic data. Use dynamic cache content. Not yet protected by the lock. Sahred memory for the cache. Cache locking Uncrustify

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. Two new configuration settings have been introduced: - `metrics_cache_max_age`, expressed in seconds, defines for how long a cached response can be served without having to compute a new one set of metrics. This parameter is required for the caching machinery to be enabled. - `metrics_cache_max_size` allows the user to avoid caching of responses greater than the expressed value in bytes. In any case, a cache response is tied to 1MegaByte, by a value set in the source code. If the cache overflows the 1MB memory limit, it is immediatly marked as invalid, that is the system is not going to serve the cached result. However, as soon as a new Prometheus request arrives, the cache will be re-filled from scratch, so if the current answer is lower than 1MB and lower than the optional `metrics_cache_max_size`, the response will be cached. Whenever an overflow of the cache happens, the user is warned. Due to the nature the cache is populated, the user could see more than one single advice in logs (at level DEBUG). The `struct prometheus` has been expanded with an inner struct, named `struct prometheus_cache` that holds the cache payload, a lock (for concurrency), the size of payload, and the timestamp the cache will be valid until. New utility functions have been added: - `metrics_cache_append(char* data)` that appends the data to the cache in the case it is safe to do it, that is no overflow in the cache size. In case the data to append makes the cache to overflow, the cache is invalidated; - `metrics_cache_finalize(void)` set the timestamp for the cache vlidation. - `metrics_cache_invalidate(void)` set the cache as invalid by zero-filling the data. Used for example when an overflow is detected. - `metrics_cache_size_to_alloc(void)` computes the size, in bytes, required for the cache payload allocation. The size is set to the default, or `metrics_cache_max_size` if configured and in any case nothing more than the max size of 1MB. - `metrics_cache_alloc(struct prometheus* prometheus)` allocates the memory for the cache and sets the initial values. This has to be called when the `struct prometheus` is allocated and initialized. The `metrics_page()` method has been refactored to check if the cache is valid, in such case the response is served directly out of cache. If the cache is not valid, the function gets the cache lock and invokes several times the append function (in inner method calls). Last, it finalizes the cache and release the lock. If the optional configuration parameter `metrics_cache_max_size` changes, the system does not reallocate memory, rather it issue a warning about the need for a restart. The documentation has been updated to reflect changes. Close agroal#219

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. Two new configuration settings have been introduced: - `metrics_cache_max_age`, expressed in seconds, defines for how long a cached response can be served without having to compute a new one set of metrics. This parameter is required for the caching machinery to be enabled. - `metrics_cache_max_size` allows the user to avoid caching of responses greater than the expressed value in bytes. In any case, a cache response is tied to 1MegaByte, by a value set in the source code. If the cache overflows the 1MB memory limit, it is immediatly marked as invalid, that is the system is not going to serve the cached result. However, as soon as a new Prometheus request arrives, the cache will be re-filled from scratch, so if the current answer is lower than 1MB and lower than the optional `metrics_cache_max_size`, the response will be cached. Whenever an overflow of the cache happens, the user is warned. Due to the nature the cache is populated, the user could see more than one single advice in logs (at level DEBUG). The cache is built as a shared memory segment (see agroal#256 and in particular <agroal#256 (comment)>). The cache is implemented by means of the `struct prometheus_cache` that has a dynamically allocated (`mmap()`) segment to handle the payload. This means that the `metrics_cache_max_size` parameter will drive the amount of memory dynamically allocated to be used as a cache. New static utility functions have been added: - `metrics_cache_append(char* data)` that appends the data to the cache in the case it is safe to do it, that is no overflow in the cache size. In case the data to append makes the cache to overflow, the cache is invalidated; - `metrics_cache_finalize(void)` set the timestamp for the cache vlidation. - `metrics_cache_invalidate(void)` set the cache as invalid by zero-filling the data. Used for example when an overflow is detected. - `metrics_cache_size_to_alloc(void)` computes the size, in bytes, required for the cache payload allocation. The size is set to the default, or `metrics_cache_max_size` if configured and in any case nothing more than the max size of 1MB. New "public" utility function: - `pgagroal_init_prometheus_cache()` will initialize the cache shared memory according to the user configuration and the result of `metrics_cache_size_to_alloc()`. The `metrics_page()` method has been refactored to check if the cache is valid, in such case the response is served directly out of cache. If the cache is not valid, the function gets the cache lock and invokes several times the append function (in inner method calls). Last, it finalizes the cache and release the lock. If the optional configuration parameter `metrics_cache_max_size` changes, the system does not reallocate memory, rather it issue a warning about the need for a restart. The documentation has been updated to reflect changes. Close agroal#219

This commit refactors the way the Prometheus metrics are served in order to provide the capability to cache the responses for a specified amount of time. Two new configuration settings have been introduced: - `metrics_cache_max_age`, expressed in seconds, defines for how long a cached response can be served without having to compute a new one set of metrics. This parameter is required for the caching machinery to be enabled. - `metrics_cache_max_size` allows the user to avoid caching of responses greater than the expressed value in bytes. In any case, a cache response is tied to 1MegaByte, by a value set in the source code. If the cache overflows the 1MB memory limit, it is immediatly marked as invalid, that is the system is not going to serve the cached result. However, as soon as a new Prometheus request arrives, the cache will be re-filled from scratch, so if the current answer is lower than 1MB and lower than the optional `metrics_cache_max_size`, the response will be cached. Whenever an overflow of the cache happens, the user is warned. Due to the nature the cache is populated, the user could see more than one single advice in logs (at level DEBUG). The cache is built as a shared memory segment (see agroal#256 and in particular <agroal#256 (comment)>). The cache is implemented by means of the `struct prometheus_cache` that has a dynamically allocated (`mmap()`) segment to handle the payload. This means that the `metrics_cache_max_size` parameter will drive the amount of memory dynamically allocated to be used as a cache. New static utility functions have been added: - `metrics_cache_append(char* data)` that appends the data to the cache in the case it is safe to do it, that is no overflow in the cache size. In case the data to append makes the cache to overflow, the cache is invalidated; - `metrics_cache_finalize(void)` set the timestamp for the cache vlidation. - `metrics_cache_invalidate(void)` set the cache as invalid by zero-filling the data. Used for example when an overflow is detected. - `metrics_cache_size_to_alloc(void)` computes the size, in bytes, required for the cache payload allocation. The size is set to the default, or `metrics_cache_max_size` if configured and in any case nothing more than the max size of 1MB. New "public" utility function: - `pgagroal_init_prometheus_cache()` will initialize the cache shared memory according to the user configuration and the result of `metrics_cache_size_to_alloc()`. The `metrics_page()` method has been refactored to immediatly acquire the cache lock. Then, if the cache is available (i.e., lock acquired), the method checks if the cache has some content, and then serves it. Otherwise, a new response is provided and built and cached. If the optional configuration parameter `metrics_cache_max_size` changes, the system does not reallocate memory, rather it issue a warning about the need for a restart. The documentation has been updated to reflect changes. Close agroal#219

jesperpedersen added enhancement Improvement to an existing feature community Issue that can be picked up by the community feature New feature and removed enhancement Improvement to an existing feature labels Apr 11, 2022

This was referenced May 30, 2022

[#219] Provide Prometheus metrics cache #256

Closed

Refactor configuration age parsing to provide support for other configuration options #257

Closed

fluca1978 mentioned this issue Jun 9, 2022

[#219] Provide Prometheus metrics cache #262

Closed

jesperpedersen closed this as completed in 98b2ed5 Jun 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Option to cache Prometheus response #219

Option to cache Prometheus response #219

jesperpedersen commented Apr 11, 2022

fluca1978 commented May 26, 2022

jesperpedersen commented May 26, 2022

Option to cache Prometheus response #219

Option to cache Prometheus response #219

Comments

jesperpedersen commented Apr 11, 2022

fluca1978 commented May 26, 2022

jesperpedersen commented May 26, 2022