Kubernetes credentials can only be refreshed once per Kubernetes API object, and can eventually expire, leading to Unauthorized errors from the Kubernetes API

[adamnovak]

See kubernetes-client/python#741

The Kubernetes module establishes some credentials at startup but never refreshes them during the life of the program.

When the credentials expire, (and we start getting 401 Unauthorized-s), we need to call load_kube_config() again (if that's how we got our credentials) and replace all the API clients.

Alternately, we could re-load the config and re-make the client at every use of Kubernetes, or just reload the config "sufficiently often" and remake the clients on every use, or something.

@glennhickey is seeing problems due to this (I think) on our Kubernetes cluster after an hour.

A workaround is running with service account credentials, which don't expire.

┆Issue is synchronized with this Jira Task
┆Issue Number: TOIL-460