[receiver/kubeletstats] Migrate kubeletstatsreceiver to the new Metrics Builder

[TylerHelmuth]

Description:

Updates kubeletstatsreceiver to use the metrics builder. Attempted to re-use as many existing conditionals as possible to ensure all functionality is the same as before. Would've like to remove the labels handling from internal/kubelet/metadata.go in favor of ResourceOptions, but there was a check for the persistent volume claim volume type in internal/kubelet/resource.go that ruined that idea. To leave that check in place, I kept internal/kubelet/metadata.go as is and opted to translate labels to ResourceOptions in internal/kubelet/resource.go.

Link to tracking Issue:

Resolves #7143

Testing:

Updated existing tests where needed. I did not add any new tests as there should be no new scenarios. All outputs should be the same as before.

[dmitryax]

Would've like to remove the labels handling from internal/kubelet/metadata.go in favor of ResourceOptions, but there was a check for the persistent volume claim volume type in internal/kubelet/resource.go that ruined that idea

I submitted a small refactor to unblock this, PTAL: #9745

[dmitryax]

Please add a changelog entry as well

[dmitryax]

@TylerHelmuth #9745 is merged, please rebase

[TylerHelmuth]

@dmitryax I removed all the resource label stuff, but the changes were a little more wide spread than just metadata.go. If the PR is too big now, I can revert the change and this PR can handle only the new metrics builder (handling resource labels to ResourceOption conversion with that switch statement I had before), and a second PR could deal with the resource labels.

[dmitryax]

It looks very nice! Just nits about removed newlines

[dmitryax]

Also I tried this build, and looks like it doesn't produce any data. Not sure what's wrong looking at the code

[TylerHelmuth]

Also I tried this build, and looks like it doesn't produce any data. Not sure what's wrong looking at the code

I will take a look.

[dmitryax]

Did you check if it produces data now?

[TylerHelmuth]

Did you check if it produces data now?

Still working towards that.

[TylerHelmuth]

Did you check if it produces data now?

Still working towards that.

So I can confirm the api calls are returning the expected data, but after the call to MetricsData, r.mb.Emit() returns an empty pmetrics.Metrics. Still investigating why.

[TylerHelmuth]

Still investigating why.

Still trying to debug this issue. Strangely, if I move accumulation to scraper.go, metrics builder is still emitting nothing:

func (r *kubletScraper) scrape(context.Context) (pmetric.Metrics, error) {
	summary, err := r.statsProvider.StatsSummary()
	r.logger.Info("summary retrieved", zap.Any("kubeletstats", summary))
	if err != nil {
		r.logger.Error("call to /stats/summary endpoint failed", zap.Error(err))
		return pmetric.Metrics{}, err
	}
	
	r.logger.Info("Preparing metrics")

	currentTime := pcommon.NewTimestampFromTime(time.Now())
	if summary.Node.Memory != nil {
		if summary.Node.Memory.AvailableBytes != nil {
			r.mb.RecordK8sNodeMemoryAvailableDataPoint(currentTime, int64(*summary.Node.Memory.AvailableBytes))
			r.logger.Info("RecordK8sNodeMemoryAvailableDataPoint")
		}
		if summary.Node.Memory.UsageBytes != nil {
			r.mb.RecordK8sNodeMemoryUsageDataPoint(currentTime, int64(*summary.Node.Memory.UsageBytes))
			r.logger.Info("RecordK8sNodeMemoryUsageDataPoint")
		}
		if summary.Node.Memory.RSSBytes != nil {
			r.mb.RecordK8sNodeMemoryRssDataPoint(currentTime, int64(*summary.Node.Memory.RSSBytes))
			r.logger.Info("RecordK8sNodeMemoryRssDataPoint")
		}
		if summary.Node.Memory.WorkingSetBytes != nil {
			r.mb.RecordK8sNodeMemoryWorkingSetDataPoint(currentTime, int64(*summary.Node.Memory.WorkingSetBytes))
			r.logger.Info("RecordK8sNodeMemoryWorkingSetDataPoint")
		}
		if summary.Node.Memory.PageFaults != nil {
			r.mb.RecordK8sNodeMemoryPageFaultsDataPoint(currentTime, int64(*summary.Node.Memory.PageFaults))
			r.logger.Info("RecordK8sNodeMemoryPageFaultsDataPoint")
		}
		if summary.Node.Memory.MajorPageFaults != nil {
			r.mb.RecordK8sNodeMemoryMajorPageFaultsDataPoint(currentTime, int64(*summary.Node.Memory.MajorPageFaults))
			r.logger.Info("RecordK8sNodeMemoryMajorPageFaultsDataPoint")
		}
	}

	//kubelet.PrepareMetricsData(r.logger, summary, metadata, r.metricGroupsToCollect, r.mb)
	data := r.mb.Emit(metadata.WithK8sNodeName(summary.Node.NodeName))
	r.logger.Info("metric prepared", zap.Any("kubeletstats", data))
	r.logger.Info("metric count", zap.Any("kubeletstats", data.MetricCount()))
	r.logger.Info("metric data point count", zap.Any("kubeletstats", data.DataPointCount()))
	r.logger.Info("resource metrics", zap.Any("kubeletstats", data.ResourceMetrics()))
	return data, nil
}

@dmitryax any ideas? I thought maybe the metrics were disabled in metadata.yaml, but it looks right to me.

I will continue investigating tomorrow.

[dmitryax]

I know why. Looks like you missed setting default value for Config.Metrics in receiver/kubeletstatsreceiver/config.go. All the metrics are just disabled

[TylerHelmuth]

I know why. Looks like you missed setting default value for Config.Metrics in receiver/kubeletstatsreceiver/config.go. All the metrics are just disabled

🤦

[TylerHelmuth]

@dmitryax added default metrics settings and reverted to the metrics slice accumulation that was happening before. Added a new struct to handle the different metrics builders. Deployed to GKE and confirmed data.