metrics: Expose process_cpu_seconds_total as a float #754

olix0r · 2020-11-27T15:59:55Z

Prometheus handles all values as f64, but we only expose values as
whole integers. This means that the process_cpu_seconds_total metric
only exposes whole second values, while Linux exposes process time in
10ms increments.

This change modifies the Counter metric type to store an additional
marker that provides a strategy for converting the stored u64 value to
f64 for export. This strategy is employed so that we can continue to
use AtomicU64 to back counters and only use floats at export-time. By
default the unit type is used to convert counters as before, but an
alternate MillisAsSeconds strategy is used to expose fractional
seconds from a millisecond counter.

This necessitates changing the histogram buckets to floats as well.
While this change doesn't modify the bucket values, this sets up future
changes to latency metrics.

Prometheus handles all values as `f64`, but we only expose values as whole integers. This means that the `process_cpu_seconds_total` metric only exposes whole second values, while Linux exposes process time in 10ms increments. This change modifies the `Counter` metric type to store an additional marker that provides a strategy for converting the stored `u64` value to `f64` for export. This strategy is employed so that we can continue to use `AtomicU64` to back counters and only use floats at export-time. By default the unit type is used to convert counters as before, but an alternate `MillisAsSeconds` strategy is used to expose fractional seconds from a millisecond counter. This necessitates changing the histogram buckets to floats as well. While this change doesn't modify the bucket values, this sets up future changes to latency metrics.

hawkw

lgtm! i commented on some nits.

hawkw · 2020-11-30T19:54:01Z

linkerd/app/core/src/telemetry/process.rs

+            if clock_ticks_per_sec != 100 {
+                warn!(
+                    clock_ticks_per_sec,
+                    ms_per_tick, "Unexpected value; process_cpu_seconds_total may be inaccurate."


👍 for warning about this (but, kinda weird formatting here... did rustfmt do that?)

yeah. rustfmt be rustfmting

hawkw · 2020-11-30T20:01:26Z

linkerd/metrics/src/counter.rs

    pub fn incr(&self) {
-        self.add(1)
+        self.add(1u64)


huh, i'm surprised the explicit type is necessary here...

Cruft from an intermediate state where add took impl Into<u64> -- no longer the case so removed the type.

hawkw · 2020-11-30T20:10:02Z

linkerd/metrics/src/lib.rs

+
+impl Factor for () {
+    fn factor(n: u64) -> f64 {
+        n.wrapping_rem(MAX_PRECISE_UINT64 + 1) as f64


i was kind of wondering if the wrapping module by the max precise value could be factored out so that Factor impls don't have to be aware of it...i suppose we could factor the max precise value too and do the modulo in Counter::value.

this is probably not worth the effort.

The intent for doing this inside of factor is so that we can adjust the precision based on the scale of the value:

https://github.com/linkerd/linkerd2-proxy/pull/754/files#diff-36a73a9a9de1e34abac85003560b0aea70e9ebb0f1cc29af3dce703bfbef7a89R56

Now that I look at this again, though, it's probably not really correct; since we may lose some of the precision before scaling it.

Actually, the current behavior seems fine

This release removes a potential panic: it was assumed that looking up a socket's peer address was infallible, but in practice this call can fail when a host is under high load. Now these failure only impact the connection-level task and not the whole proxy proces. Also, the `process_cpu_seconds_total` metric is now exposed as a float so that its value may include fractional seconds with 10ms granularity. --- * io: Make peer_addr fallible (linkerd/linkerd2-proxy#755) * metrics: Expose process_cpu_seconds_total as a float (linkerd/linkerd2-proxy#754)

This release removes a potential panic: it was assumed that looking up a socket's peer address was infallible, but in practice this call can fail when a host is under high load. Now these failures only impact the connection-level task and not the whole proxy proces. Also, the `process_cpu_seconds_total` metric is now exposed as a float so that its value may include fractional seconds with 10ms granularity. --- * io: Make peer_addr fallible (linkerd/linkerd2-proxy#755) * metrics: Expose process_cpu_seconds_total as a float (linkerd/linkerd2-proxy#754)

@hodbn

* proxy: v2.119.0 (#5200) This release modifies the default idle timeout to 5s for outbound clients and 20s for inbound clients. This prevents idle clients from consuming memory at the cost of performing more discovery resolutions for periodic but infrequent traffic. This is intended to reduce the proxy's memory footprint, especially on Prometheus instances. The proxy's *ring* and rustls dependencies have also been updated. --- * Update *ring* and rustls dependencies (linkerd/linkerd2-proxy#735) * http: Configure client connection pools (linkerd/linkerd2-proxy#734) * cli: Remove get cmd and relevant tests (#5202) Fixes #5190 `linkerd get` is not used currently and works only for pods. This can be removed instead as per the issue. This branch removes the command and also the associated unit and integration tests. Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com> * cli: remove logs subcommand and tests (#5203) Fixes #5191 The logs command adds a external dependency that we forked to work but does not fit within linkerd's core set of responsibilities. Hence, This is being removed. For capabilities like this, The Kubernetes plugin ecosystem has better and well maintained tools that can be used. Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com> * Remove logs comparisons in integration tests (#5223) The rare cases where these tests were useful don't make up for the burden of maintaing them, having different k8s version change the messages and having unexpected warnings come up that didn't affect the final convergence of the system. With this we also revert the indirection added back in #4538 that fetched unmatched warnings after a test had failed. * Add endpoint to GetProfile response (#5227) Context: #5209 This updates the destination service to set the `Endpoint` field in `GetProfile` responses. The `Endpoint` field is only set if the IP maps to a Pod--not a Service. Additionally in this scenario, the default Service Profile is used as the base profile so no other significant fields are set. ### Examples ``` # GetProfile for an IP that maps to a Service ❯ go run controller/script/destination-client/main.go -method getProfile -path 10.43.222.0:9090 INFO[0000] fully_qualified_name:"linkerd-prometheus.linkerd.svc.cluster.local" retry_budget:{retry_ratio:0.2 min_retries_per_second:10 ttl:{seconds:10}} dst_overrides:{authority:"linkerd-prometheus.linkerd.svc.cluster.local.:9090" weight:10000} ``` Before: ``` # GetProfile for an IP that maps to a Pod ❯ go run controller/script/destination-client/main.go -method getProfile -path 10.42.0.20 INFO[0000] retry_budget:{retry_ratio:0.2 min_retries_per_second:10 ttl:{seconds:10}} ``` After: ``` # GetProfile for an IP that maps to a Pod ❯ go run controller/script/destination-client/main.go -method getProfile -path 10.42.0.20 INFO[0000] retry_budget:{retry_ratio:0.2 min_retries_per_second:10 ttl:{seconds:10}} endpoint:{addr:{ip:{ipv4:170524692}} weight:10000 metric_labels:{key:"control_plane_ns" value:"linkerd"} metric_labels:{key:"deployment" value:"fast-1"} metric_labels:{key:"pod" value:"fast-1-5cc87f64bc-9hx7h"} metric_labels:{key:"pod_template_hash" value:"5cc87f64bc"} metric_labels:{key:"serviceaccount" value:"default"} tls_identity:{dns_like_identity:{name:"default.default.serviceaccount.identity.linkerd.cluster.local"}} protocol_hint:{h2:{}}} ``` Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com> * cli: Fix custom namespace installation (#5241) The `--linkerd-namespace` flag was not honored by the `install` command. This change updates the install templating to use the value of this flag. * cli: Don't check for SAN in root and intermediate certs (#5237) As discussed in #5228, it is not correct for root and intermediate certs to have SAN. This PR updates the check to not verify the intermediate issuer cert with the identity dns name (which checks with SAN and not CN as the the `verify` func is used to verify leaf certs and not root and intermediate certs). This PR also avoids setting a SAN field when generating certs in the `install` command. Fixes #5228 * proxy: v2.121.0 (#5253) This release changes error handling to teardown the server-side connection when an unexpected error is encountered. Additionally, the outbound TCP routing stack can now skip redundant service discovery lookups when profile responses include endpoint information. Finally, the cache implementation has been updated to reduce latency by removing unnecessary buffers. --- * h2: enable HTTP/2 keepalive PING frames (linkerd/linkerd2-proxy#737) * actions: Add timeouts to GitHub actions (linkerd/linkerd2-proxy#738) * outbound: Skip endpoint resolution on profile hint (linkerd/linkerd2-proxy#736) * Add a FromStr for dns::Name (linkerd/linkerd2-proxy#746) * outbound: Avoid redundant TCP endpoint resolution (linkerd/linkerd2-proxy#742) * cache: Make the cache cloneable with RwLock (linkerd/linkerd2-proxy#743) * http: Teardown serverside connections on error (linkerd/linkerd2-proxy#747) * Check correct label value when setting protocl hint (#5267) This fixes an issue where the protocol hint is always set on endpoint responses. We now check the right value which determines if the pod has the required label. A test for this has been added to #5266. Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com> * Add safe accessor for Global in linkerd-config (#5269) CLI crashes if linkerd-config contains unexpected values. Add a safe accessor that initializes an empty Global on the first access. Refactor all accesses to use the newly introduced accessor using gopls. Add test for linkerd-config data without Global. Fixes #5215 Co-authored-by: Itai Schwartz <yitai27@gmail.com> Signed-off-by: hodbn <hodbn@users.noreply.github.com> * proxy: v2.122.0 (#5279) This release addresses some issues reported around clients seeing max-concurrency errors by increasing the default in-flight request limit to 100K pending requests. Additionally, the proxy now sets an appropriate content-type when synthesizing gRPC error responses. --- * style: fix some random clippy lints (linkerd/linkerd2-proxy#749) * errors: Set `content-type` for synthesized grpc errors (linkerd/linkerd2-proxy#750) * concurrency-limit: Drop permit on readiness (linkerd/linkerd2-proxy#751) * Increase the default buffer capacity to 100K (linkerd/linkerd2-proxy#752) * Change default max-in-flight and buffer-capacity (linkerd/linkerd2-proxy#753) * proxy: v2.123.0 (#5301) This release removes a potential panic: it was assumed that looking up a socket's peer address was infallible, but in practice this call can fail when a host is under high load. Now these failures only impact the connection-level task and not the whole proxy proces. Also, the `process_cpu_seconds_total` metric is now exposed as a float so that its value may include fractional seconds with 10ms granularity. --- * io: Make peer_addr fallible (linkerd/linkerd2-proxy#755) * metrics: Expose process_cpu_seconds_total as a float (linkerd/linkerd2-proxy#754) * Release notes for stable-2.9.1 ## stable-2.9.1 This stable release contains a number of proxy enhancements: better support for high-traffic workloads, improved performance by eliminating unnecessary endpoint resolutions for TCP traffic and properly tearing down serverside connections when errors occur, and reduced memory consumption on proxies which maintain many idle connections (such as Prometheus' proxy). On the CLI and control plane sides, it relaxes checks on root and intermediate certificates (following X509 best practices), and fixes two issues: one that prevented installation of the control plane into a custom namespace and one which failed to update endpoint information when a headless service was modified. * Proxy: * Addressed some issues reported around clients seeing max-concurrency errors by increasing the default in-flight request limit to 100K pending requests * Reduced the default idle connection timeout to 5s for outbound clients and for inbound clients to reduce the proxy's memory footprint, especially on Prometheus instances * Fixed an issue where the proxy did not receive updated endpoint information when a headless service was modified * Added HTTP/2 keepalive PING frames * Removed logic to avoid redundant TCP endpoint resolution * Fixed an issue where serverside connections were not torn down when an error occurred * CLI / Control Plane: * Fixed a CLI issue where the `linkerd-namespace` flag was not honored when passed to the `install` and `upgrade` commands * Updated `linkerd check` so that it doesn't attempt to validate the subject alternative name (SAN) on root and intermediate certificates. SANs for leaf certificates will continue to be validated * Fixed an issue in the destination service where endpoints always included a protocol hint, regardless of the controller label being present or not * Removed the `get` and `logs` command from the CLI * No longer panic in rare cases when `linkerd-config` doesn't have an entry for `Global` configs (thanks @hodbn!) * proxy: v2.124.0 (#5323) This release updates the proxy's `*ring*` dependency to pick up the latest changes from BoringSSL. Additionally, we've audited uses of non-cryptographic random number generators in the proxy to ensure that each balancer/router intializes its own RNG state. --- * Audit uses of SmallRng (linkerd/linkerd2-proxy#757) * Update *ring* to 0.6.19 (linkerd/linkerd2-proxy#758) * metrics: Support the Summary metric type (linkerd/linkerd2-proxy#756) Co-authored-by: Oliver Gould <ver@buoyant.io> Co-authored-by: Tarun Pothulapati <tarunpothulapati@outlook.com> Co-authored-by: Kevin Leimkuhler <kevin@kleimkuhler.com> Co-authored-by: hodbn <hodbn@users.noreply.github.com> Co-authored-by: Itai Schwartz <yitai27@gmail.com>

olix0r requested a review from a team November 27, 2020 15:59

hawkw approved these changes Nov 30, 2020

View reviewed changes

olix0r added 2 commits November 30, 2020 20:21

Remove cruft

9b4d8f5

Broaden Default impl

28aa0a4

olix0r merged commit 876ae02 into main Nov 30, 2020

olix0r deleted the ver/metrics-factor branch November 30, 2020 23:38

olix0r mentioned this pull request Dec 1, 2020

proxy: v2.123.0 linkerd/linkerd2#5301

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metrics: Expose process_cpu_seconds_total as a float #754

metrics: Expose process_cpu_seconds_total as a float #754

olix0r commented Nov 27, 2020

hawkw left a comment

hawkw Nov 30, 2020

olix0r Nov 30, 2020

hawkw Nov 30, 2020

olix0r Nov 30, 2020

hawkw Nov 30, 2020

olix0r Nov 30, 2020

olix0r Nov 30, 2020

metrics: Expose process_cpu_seconds_total as a float #754

metrics: Expose process_cpu_seconds_total as a float #754

Conversation

olix0r commented Nov 27, 2020

hawkw left a comment

Choose a reason for hiding this comment

hawkw Nov 30, 2020

Choose a reason for hiding this comment

olix0r Nov 30, 2020

Choose a reason for hiding this comment

hawkw Nov 30, 2020

Choose a reason for hiding this comment

olix0r Nov 30, 2020

Choose a reason for hiding this comment

hawkw Nov 30, 2020

Choose a reason for hiding this comment

olix0r Nov 30, 2020

Choose a reason for hiding this comment

olix0r Nov 30, 2020

Choose a reason for hiding this comment