pkg/kube-metrics: Add initial operator specific metrics

[lilic]

Operator specific metrics are generated based on the api/kind given
deployed in the given namespace(s).

See https://github.com/operator-framework/operator-sdk/blob/master/doc/proposals/metering-operator-metrics.md for the full background on this work.

Things to be done:

• kube-state-metrics has not been released yet, so currently pinning this to master, so we should wait until that is released so we can pin. But there should not be any major changes from master until the next release so this is ready for review and feedback.
• Currently we detect the namespace in which the operator is deployed in and use that by default along with the namespaces passed by the user. In the future we might want to also detect in which namespace all the CRs are deployed in, if the operator is cluster scoped. Not sure if we want this done in the current PR? Since everything will be done in the kube-metrics package and will not be user facing I think it can be done at a later point.
• Expose the metrics port in the Service file. This includes breaking changes and would prefer to do it in another PR as this one large enough to review.
• Update proposal to done once this is merged.

Closes #616

[lilic]

This is ready for review, the WIP can be removed when the new kube-state-metrics release happens. :)

[estroz]

Looks good overall, but a few nits and style suggestions need to be addressed.

[lilic]

Done @estroz cam you PTAL, thanks!

[joelanford]

Since setting up metrics goes hand-in-hand with adding APIs, can we structure the scaffolding and registration of the metrics the same way that we structure the scaffolding and scheme building of the APIs?

I think that would mean:

1. When operator-sdk new is called:
   • Create pkg/metrics/metrics.go, similar to pkg/apis/apis.go
   • In cmd/manager/main.go, use what's been setup via the exposed vars/functions in pkg/metrics/metrics.go in the call to ServeOperatorSpecificMetrics(). Since the API-specific stuff would be handled elsewhere, I think ServeOperatorSpecificMetrics could be made generic and included in the SDK.
2. When operator-sdk add api is called:
   • Create pkg/metrics/addtometrics_group_version.go, similar to pkg/apis/addtoscheme_group_version.go
   • Create pkg/metrics/group/version/register.go, similar to pkg/apis/group/version/register.go
   • Create pkg/metrics/group/version/kind_metrics.go, similar to pkg/apis/group/version/kind_types.go

I think this approach would solve the issues I mentioned in the review, and it would align better with how we scaffold and register other project-specific resources.

For the namespace question, can we pass a slice of namespaces into ServeOperatorSpecificMetrics (to be forward-compatible with upcoming multi-namespace support), and for now use what we get from k8sutil.GetWatchNamespace()?

Lastly (and this may have been discussed before because it sounds familiar, so forgive me), is it possible/does it make sense to hook these metrics into the controller-runtime metrics server so we don't have to run two servers on two ports?

[lilic]

Lastly (and this may have been discussed before because it sounds familiar, so forgive me), is it possible/does it make sense to hook these metrics into the controller-runtime metrics server so we don't have to run two servers on two ports?

No, because these metrics don't use the prometheus client_golang library we can't do that right now.