Prometheus monitoring test

[sekka1]

We recently did an update where we made some change to the kubelet to make it more "secure".

Kops changes:

  kubelet:
    # https://github.com/kubernetes/kops/blob/master/docs/security.md#kubelet-api
    anonymousAuth: false
    tlsMinVersion: VersionTLS12
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_RSA_WITH_AES_256_GCM_SHA384

As usual security is hard and it gets tangled in many different things.

After the update, everything seems to work fine but then after a Prometheus alert came by:

Kubelet has disappeared from Prometheus target discovery.
Annotations:
    - message: Kubelet has disappeared from Prometheus target discovery.
    - runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletdown
   
   Details:
    - alertname = KubeletDown
    - cluster = dev
    - prometheus = monitoring/prometheus-operator-prometheus
    - severity = critical

Looking into this in prometheus:

The prometheus service monitor is getting a 401 now anauthorized. The update broke some monitoring thing.

We should be able to leverage Prometheus monitoring to track if the updates broke anything or the stats like cpu/memory usage changes significantly after the update.

I suppose the testing will have to monitor this cluster for a bit (1 hour at the least?)? An then tell use this is something new.