KEP 2258: Node service log viewer

[aravindhp]

KEP to introduce kubectl option for viewing logs of system services on Windows and Linux nodes.

Co-authored-by: Christian Glombek cglombek@redhat.com

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Welcome @aravindhp!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @aravindhp. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aravindhp]

/cc @immuzz

[aravindhp]

I signed it

[marosset]

/sig node
Since this has proposed kubelet changes

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
• Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[marosset]

/check-cla

[aravindhp]

I signed it

[ddebroy]

This is a great enhancement! I realize it's WIP but had a few early comments.

[ddebroy]

I guess this is obvious but maybe you can point out here that reporting logs for nodes/kubelets that have config/connection issues with the cluster is out-of-scope.

[aravindhp]

Done

[ddebroy]

Windows kubelets may log to just a regular file. Would there be some logic here to figure out whether the log type configured and stream them out based on the configuration?

[LorbusChris]

I don't think we've planned for any detection logic for this case.
One thing to note is that if the kubelet (or any other process) logs to C:\var\log\somepath on Windows, those log files can be streamed by the kubelet's existing log server, and one part of this proposal is implementing a client command for it.

[aravindhp]

@ddebroy I have sort of addressed this in my next commit. PTAL.

[ddebroy]

Given the kubectl enhancements, I guess sig-cli needs to be involved as well and needs to approve?

[marosset]

We probably want someone form sig-auth to approve here as well.
At least from the windows side exposing the ability to query arbitrary event logs can expose a lot more system information than what is exposed in kubelet logs.

[aravindhp]

Added sig-auth to participating sigs.

[aravindhp]

/sig cli

[smarterclayton]

I don't see a clear principle that delineates this feature from "we made a general log-query API". How far is too far and why ?

Does Kubelet know about the node? Yes
Does Kubelet have a rough idea of an opinion of the shape of logging of process it runs on the node? Yes
Does Kubelet have a goal of making debugging workloads on the node? Yes
Does Kubelet have a goal of working well on Linux and Windows machines such that admins can correlate failures outside the Kubelet but inside the Kubernetes system? 75% yes

<---

Does Kubelet own the machine? No
Should individual distributions of Kubernetes / services be required to be conformant on how to troubleshoot the node? Maybe not?
Would we allow a lot of other types of Linux systems to inject this same logging interface? Maybe not?

For OpenShift all of the answers turned out to be yes, and so it was a win to add something like this (because we explicitly took choice about node configuration away). I don't think that's universal. I don't think this endpoint should be required for conformance. Would most Kube distros have this, find value in it, and lead to happier and healthy clusters, without requiring us to solve completely? Probably.

I remember us having this same discussion about whether pod logs should be viewable, but we had the bar of Docker log to contend with, so we just went ahead and did it. Same for exec - exec was so useful we decided not to fallback on SSH and just put exec in. I can't imagine Kubernetes without pod logs and pod exec.

[thockin]

For OpenShift all of the answers turned out to be yes, and so it was a win to add something like this (because we explicitly took choice about node configuration away).

Agree, and I think it would be a win for most installations. I am just trying to find the edges of it.

I don't think this endpoint should be required for conformance. Would most Kube distros have this, find value in it, and lead to happier and healthy clusters, without requiring us to solve completely? Probably.

I think I agree with that.

My only remaining concern is forcing users to know and spell out files vs journals - can we find some way to help that?

[immuzz]

@aravindhp Thank you for driving this KEP. This is one of the most asked feature by Windows Containers customers for scenarios like gMSA where they have to individually log into each node to view logs just to set it up. I have reviewed the KEP thoroughly and have asked Windows engineers to review as well. Look good to us. This will be very useful in making K8s more managed. Cant wait to try this and happy to help along the way! Thanks again!

[jayunit100]

... FWIW we'd take advantage of this if it was available :)

[aravindhp]

My only remaining concern is forcing users to know and spell out files vs journals - can we find some way to help that?

@thockin, I can't think of an easy way to do this with making assumptions about the service and how it would log. Moreover given this feature is restricted to cluster admins and is advanced, won't having knowledge of whether they should asking for a file or journal be fine? Especially given they need this knowledge today when they have to log into the node and inspect it.

[thockin]

@aravindhp

I can't think of an easy way to do this with making assumptions about the service and how it would log

Why can't this be an abstraction? If I ask for kubectl logs nodes foobar have kubelet first look for a "foobar" journal, then look under "/var/log" for "foobar" files and heuristically choose which one to serve.

Why push this on users?

[aravindhp]

@thockin

If I ask for kubectl logs nodes foobar have kubelet first look for a "foobar" journal, then look under "/var/log" for "foobar" files and heuristically choose which one to serve

This was the assumption I was hesitant to make i.e. a service named foobar implies that it will log to /var/log/foobar* or /var/log/foobar/*. Are you suggesting that we do a best effort but still have the option of being granular for services like foobar that log to /var/log/barfoo?

Also services like kube-proxy logs to multiple files like kube-proxy.exe.ERROR, kube-proxy.exe.INFO. In those cases do we output all the files?

[marosset]

To me, the biggest value that this enhancement adds to K8s is that it makes it possible to grab logs off nodes without needed to remotely connect to said nodes - this is especially important for Windows where configuring the nodes for a remote access is more complicated than adding SSH keys to a file.

Today in order to debug Windows nodes users need to first enable remote access (which is usually done by interacting with vendor specific infrastructure to manipulate the node) and then start looking for the logs they want to inspect.
This enhancement as is would solve one of these two problems.

Why can't this be an abstraction? If I ask for kubectl logs nodes foobar have kubelet first look for a "foobar" journal, then look under "/var/log" for "foobar" files and heuristically choose which one to serve.

Something like this may help but I have a small concern that this could increase complexity and cause some hinderances.
My feeling is that anyone using this feature to debug issues will most likely already know where the logs are on their nodes.

I wouldn't be opposed to adding in some heuristics or something to help in finding logs but do think it would be important to let users get at the exact logs they are looking for and/or handle cases like

Are you suggesting that we do a best effort but still have the option of being granular for services like foobar that log to /var/log/barfoo?

[thockin]

This was the assumption I was hesitant to make i.e. a service named foobar implies that it will log to /var/log/foobar* or /var/log/foobar/*. Are you suggesting that we do a best effort but still have the option of being granular for services like foobar that log to /var/log/barfoo?

Yeah, I am saying that, on average, I expect this sort of a heuristic to work pretty well. If we want to add more specific manual overrides, those are the exception, rather than the norm.

Also services like kube-proxy logs to multiple files like kube-proxy.exe.ERROR, kube-proxy.exe.INFO. In those cases do we output all the files?

Yay glog. Those are additive - the INFO file is inclusive of the ERROR file. If we want to serve both through this API, maybe we have an extra (optional) arg that factors into the heuristic.

And if I am wrong that the heuristic approach works, then I will waive further objections.

So how can we prove whether it does or doesn't work?

[aravindhp]

@thockin

Yay glog. Those are additive - the INFO file is inclusive of the ERROR file. If we want to serve both through this API, maybe we have an extra (optional) arg that factors into the heuristic.

Yay glog indeed 😃 In fact there are three files including the WARNING file which is also included in the INFO file. So in heuristic approach by default we only server the INFO file or do we serve all files starting with kube-proxy i.e. starting with the service name? Then we can have the specific file option if the user wants to look at only one of them.

And if I am wrong that the heuristic approach works, then I will waive further objections.
So how can we prove whether it does or doesn't work?

One way would be to call out the heuristic approach in the KEP. Then implement the feature without heuristics for alpah and get feedback from the users. Alternatively implement the feature with the heuristics approach and without the journal vs file options for alpha and get feedback from the users. Then react to the feedback appropriately when moving from alpha to beta. I am open to other options too.

[thockin]

in heuristic approach by default we only server the INFO file or do we serve all files

as a heuristic, if we see 3 such files, we can assume they are glog output and choose INFO. If there's really demand for only error messages, we can expand the heuristic.

My fear is that if we implement with a lot of control we will have a hard time removing them, whereas if we start with relatively little control, we may not need them at all.

I should caveat - this is armchair architecture. From my distance I can handwave about it, but I don't have to implement all the crazy suggestions. It may be that I am asking for dumb things, and I just need people to show me why.

[aravindhp]

@thockin Thank you for the patient feedback. I have updated the KEP to reflect the heuristic approach you suggested. Please take a look.

[thockin]

I'm pretty happy - small nits only from me

/lgtm

[thockin]

Why do some flags only work on some implementations? Can we at least clarify somewhere that "some capabilities are optional, and may not be supported by some sources of log information", rather than specifically "journal"?

[aravindhp]

I will add a note to call this out during the implementation. Would that suffice or should I open a follow up PR to amend the enhancement?

[thockin]

same for these

[thockin]

I'd also say that the heuristic can grow and evolve as we get real feedback.

[aravindhp]

Agreed

[aravindhp]

Thank your for the review, @thockin

[aravindhp]

I will add a note to call this out during the implementation. Would that suffice or should I open a follow up PR to amend the enhancement?

[aravindhp]

Agreed

[carlisia]

Hello @aravindhp 👋, 1.22 Docs Shadow here.
This enhancement is marked as ‘Needs Docs’ for the 1.22 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.22 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Fri July 9, 11:59 PM PDT.

Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you! 🙏