Add metrics for endpointslice state #1923

sawsa307 · 2023-01-27T01:32:50Z

Added metrics to collect the state of each endpointslice.

Example:
HELP neg_controller_neg_sync_endpoint_slice_state Current count of endpoint slices in each state
TYPE neg_controller_neg_sync_endpoint_slice_state gauge
neg_controller_neg_sync_endpoint_slice_state{endpoint_slice_label="endpointsliceTotal"} 3
neg_controller_neg_sync_endpoint_slice_state{endpoint_slice_label="endpointsliceWithDuplicateEP"} 0
neg_controller_neg_sync_endpoint_slice_state{endpoint_slice_label="endpointsliceWithMissingFieldEP"} 0
neg_controller_neg_sync_endpoint_slice_state{endpoint_slice_label="endpointsliceWithMissingNodeNameEP"} 0
neg_controller_neg_sync_endpoint_slice_state{endpoint_slice_label="endpointsliceWithMissingPodEP"} 0
neg_controller_neg_sync_endpoint_slice_state{endpoint_slice_label="endpointsliceWithMissingZoneEP"} 0

k8s-ci-robot · 2023-01-27T01:32:59Z

Hi @sawsa307. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sawsa307 · 2023-01-27T01:33:01Z

/assign @swetharepakula

aojea · 2023-01-28T22:46:26Z

pkg/flags/flags.go

@@ -256,7 +257,8 @@ L7 load balancing. CSV values accepted. Example: -node-port-ranges=80,8080,400-5
 	flag.BoolVar(&F.EnableL4ILBDualStack, "enable-l4ilb-dual-stack", false, "Enable Dual-Stack handling for L4 Internal Load Balancers")
 	flag.BoolVar(&F.EnableMultipleIGs, "enable-multiple-igs", false, "Enable using multiple unmanaged instance groups")
 	flag.IntVar(&F.MaxIGSize, "max-ig-size", 1000, "Max number of instances in Instance Group")
-	flag.DurationVar(&F.MetricsExportInterval, "metrics-export-interval", 10*time.Minute, `Period for calculating and exporting metrics related to state of managed objects.`)
+	flag.DurationVar(&F.UsageMetricsExportInterval, "usage-metrics-export-interval", 10*time.Minute, `Period for calculating and exporting metrics related to state of managed objects.`)
+	flag.DurationVar(&F.SyncMetricsExportInterval, "sync-metrics-export-interval", 5*time.Second, `Period for calculating and exporting metrics related to state of syncer.`)


just curious, why so much difference, one 10 mins and the other 5 sec

Thanks for reviewing!
The reason why we use a much smaller interval is we think 10min is too long for us to collect syncer metrics. Some errors can resolve by syncer within 10mins, and we might lose information if the export interval is too long.
In addition, we can still query and aggregate to get information for a longer period of time. It is easier to aggregate data than restore lost information.

aojea · 2023-01-28T22:48:28Z

pkg/neg/metrics/neg_metrics_collector.go

+	epCountLabel      = "ep_count"
+	epsCountLabel     = "eps_count"


ep is endpoint and eps endpointslice?

should not more readable to have the whole name instead ?

Sounds good. I'll update this. Thanks!

aojea · 2023-01-28T22:49:43Z

pkg/neg/metrics/neg_metrics_collector.go

+	metricsInterval time.Duration
+}
+
+// init registers ingress usage metrics.


to register metrics you should do it only once, see example on the other file

ingress-gce/pkg/neg/metrics/metrics.go

Lines 130 to 141 in 0f312a7

var register sync.Once

func RegisterMetrics() {

register.Do(func() {

prometheus.MustRegister(NegOperationLatency)

prometheus.MustRegister(NegOperationEndpoints)

prometheus.MustRegister(ManagerProcessLatency)

prometheus.MustRegister(SyncerSyncLatency)

prometheus.MustRegister(LastSyncTimestamp)

prometheus.MustRegister(InitializationLatency)

})

}

Thanks for the example. Updated.

Updated in #1911

aojea · 2023-01-28T22:52:23Z

/ok-to-test

Created sync metrics collector to collect sync related metrics. Added metrics to collect the cumulative count of sync results. Added export interval flag.

Added metrics to collect the current count of syncers in each status.

Added metrics to collect the state of each endpoint.

Added metrics to collect the state of each endpointslice.

k8s-ci-robot · 2023-02-02T20:01:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sawsa307
Once this PR has been reviewed and has the lgtm label, please ask for approval from swetharepakula by writing /assign @swetharepakula in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2023-02-04T05:34:54Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sawsa307 · 2023-03-22T17:39:45Z

Included in #2019

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 27, 2023

k8s-ci-robot requested review from aojea and panslava January 27, 2023 01:32

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 27, 2023

k8s-ci-robot assigned swetharepakula Jan 27, 2023

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 27, 2023

aojea reviewed Jan 28, 2023

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 28, 2023

sawsa307 added 13 commits February 1, 2023 23:17

create neg metrics collector

69d0437

create exportInterval flag

d85e3e7

initialize metrics collector

3e6ac45

Add metrics for sync result

a3f5de6

Created sync metrics collector to collect sync related metrics. Added metrics to collect the cumulative count of sync results. Added export interval flag.

define syncer statuses

06ccd3e

create syncer status metrics

fffde4e

Add metrics for syncer status

ca9d8a4

Added metrics to collect the current count of syncers in each status.

fix typo

07427b9

add ep state metrics

8d04ec5

refactor for metric collection

6093ec3

Add metrics for endpoint state

11f5710

Added metrics to collect the state of each endpoint.

add eps state metric

ae830f3

Add metrics for endpointslice state

9217179

Added metrics to collect the state of each endpointslice.

sawsa307 force-pushed the neg-eps-state-metric branch from ae3557c to 9217179 Compare February 2, 2023 20:00

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 4, 2023

sawsa307 marked this pull request as draft February 4, 2023 05:50

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 4, 2023

sawsa307 closed this Mar 22, 2023

sawsa307 deleted the neg-eps-state-metric branch April 4, 2023 00:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metrics for endpointslice state #1923

Add metrics for endpointslice state #1923

sawsa307 commented Jan 27, 2023 •

edited

Loading

k8s-ci-robot commented Jan 27, 2023

sawsa307 commented Jan 27, 2023

aojea Jan 28, 2023

sawsa307 Jan 31, 2023 •

edited

Loading

aojea Jan 28, 2023

sawsa307 Jan 30, 2023

aojea Jan 28, 2023 •

edited

Loading

sawsa307 Jan 30, 2023

sawsa307 Jan 30, 2023

aojea commented Jan 28, 2023

k8s-ci-robot commented Feb 2, 2023

k8s-ci-robot commented Feb 4, 2023

sawsa307 commented Mar 22, 2023

	var register sync.Once

	func RegisterMetrics() {
	register.Do(func() {
	prometheus.MustRegister(NegOperationLatency)
	prometheus.MustRegister(NegOperationEndpoints)
	prometheus.MustRegister(ManagerProcessLatency)
	prometheus.MustRegister(SyncerSyncLatency)
	prometheus.MustRegister(LastSyncTimestamp)
	prometheus.MustRegister(InitializationLatency)
	})
	}

Add metrics for endpointslice state #1923

Add metrics for endpointslice state #1923

Conversation

sawsa307 commented Jan 27, 2023 • edited Loading

k8s-ci-robot commented Jan 27, 2023

sawsa307 commented Jan 27, 2023

aojea Jan 28, 2023

Choose a reason for hiding this comment

sawsa307 Jan 31, 2023 • edited Loading

Choose a reason for hiding this comment

aojea Jan 28, 2023

Choose a reason for hiding this comment

sawsa307 Jan 30, 2023

Choose a reason for hiding this comment

aojea Jan 28, 2023 • edited Loading

Choose a reason for hiding this comment

sawsa307 Jan 30, 2023

Choose a reason for hiding this comment

sawsa307 Jan 30, 2023

Choose a reason for hiding this comment

aojea commented Jan 28, 2023

k8s-ci-robot commented Feb 2, 2023

k8s-ci-robot commented Feb 4, 2023

sawsa307 commented Mar 22, 2023

sawsa307 commented Jan 27, 2023 •

edited

Loading

sawsa307 Jan 31, 2023 •

edited

Loading

aojea Jan 28, 2023 •

edited

Loading