go-runner in 1.24.8 kube-proxy Docker Image is built with 1.17.3 GO

[jhawkins1]

What happened?

It appears although 1.24.x moved to 1.18 GO that the "go-runner" in kube-proxy Docker Image is built using 1.17.3 GO. This is being detected by Vulnerability Scanners, therefore, all 1.17.3 CVEs are being reported for the kube-proxy Image due to "go-runner".

What did you expect to happen?

"go-runner" should be built with the same version of GO that the Kubernetes Release is using. For 1.24.8 that is GO 1.18.x.

How can we reproduce it (as minimally and precisely as possible)?

Vulnerability Scan of the Container Image will show that 1.17.3 is being used to build the "go-runner" in the kube-proxy image.

Anything else we need to know?

No response

Kubernetes version

1.24.8

Cloud provider

N/A

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@jhawkins1: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

/remove-sig api-machinery
/sig networking

[k8s-ci-robot]

@dims: The label(s) sig/networking cannot be applied, because the repository doesn't have them.

In response to this:

/remove-sig api-machinery
/sig networking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jhawkins1]

/sig security

[jhawkins1]

/sig release

[aojea]

we'll need help from sig-release here, I tried to find out how the image is built for 1.24 without success

[jhawkins1]

Looks like go-runner was added to kube-proxy in 1.23 via kubernetes/kubernetes#106086 . I see @BenTheElder played a role with the adding of go-runner. Maybe the folks on that Issue and set of PRs could aid in tracking down this issue.

[BenTheElder]

kube-proxy doesn't use this image in 1.25+
The image is in the kubernetes/release repo

[BenTheElder]

Unfortunately we probably can't backport the 1.25 improvements since they could break people depending on other things in the image.

[jhawkins1]

@BenTheElder , looking at the build it appears it should be using 1.18 GO for the Image being generated and then utilized by Kube Proxy. Confusing to me how it is using the 1.17.3 GO... I am fuzzy though how this works -- my first time looking at the build mechanisms.

[jhawkins1]

@BenTheElder .... Ping... Any progress or further thoughts on this issue? Can the go-runner that kube-proxy is using be built with the same GO version used by 1.24.x K8 without backporting other changes for 1.25.x that you eluded to in your earlier comment? We would like to get the CVE fixes in 1.24.x without needing to move to 1.25.x in the near term -- 1.25.x is in our plans for early/mid 2023, before 1.24 goes EOL in July. Getting some pressures on the Vuln Management side to get the CVEs remediate in the nearer term.

[BenTheElder]

I'm sorry but I'm not working on this, and I'm out right now.

Can the go-runner that kube-proxy is using be built with the same GO version used by 1.24.x K8 without backporting other changes for 1.25.x that you eluded to in your earlier comment?

Maybe? This is not easy to answer, Go version upgrades often require updating dependencies and otherwise patching to work around behavior / standard library changes. You'll have to look into what changes are between these go versions.

Focus in the kube-proxy image work has been on eliminating dependencies in the image going forward for
various reasons including avoiding CVE issues. We're also working on better plans for maintaining go versions, but historically Kubernetes has not done Go minor version upgrades within patch releases for various compatibility reasons so you'd have to build your own patched image. Most distros will be maintaining this sort of thing for you and we do the best we can upstream.

cc @kubernetes/sig-release-leads

@liggitt has been working on a patch set to be able to safely upgrade minor go versions on a patch releases for the first time, but I don't think it has included go-runner, and he's also out right now.

[liggitt]

@liggitt has been working on a patch set to be able to safely upgrade minor go versions on a patch releases for the first time, but I don't think it has included go-runner, and he's also out right now.

it did bump the go-runner image for 1.23 and 1.24:

• kubernetes/kubernetes@fd427d8#diff-64833c91e011032ba2c7a5d17a52dd97ff3d10ecc644ffbfad15259718c75813R94
• kubernetes/kubernetes@f978e33#diff-64833c91e011032ba2c7a5d17a52dd97ff3d10ecc644ffbfad15259718c75813R94

but this issue is saying the go-runner version for kube-proxy is not controlled by that line? since https://github.com/kubernetes/kubernetes/blob/v1.24.8/build/common.sh#L94 also referenced go1.18.8 but this issue is saying the v1.24.8 kube-proxy image uses a go-runner built with go1.17?

[jhawkins1]

@liggitt , the Vulnerability Scanner we are using (Palo Alto Prisma) is detecting 1.17.3 for the go-runner Application (/go-runner) in the 1.24.8 kube-proxy Image (k8s.gcr.io/kube-proxy:v1.24.8). Other K8 GO Apps (ex. usr/local/bin/kube-proxy) are showing using 1.18.8. We recently upgraded to 1.24.x family of K8 and assumed 1.18.x was used for everything, and discovered it appears the GO-Runner was built with 1.17.3 GO. Confusing, thus the issue submission.

[liggitt]

hmm, looks like the version of go-runner included in the kube-proxy iptables image comes from https://github.com/kubernetes/release/blame/master/images/build/debian-iptables/Makefile#L24

that is actually stale on master (I would have expected go1.19.4), but I also don't know if/how that got bumped for the debian-iptables image version used in 1.23 and 1.24

I think this issue probably belongs in the https://github.com/kubernetes/release repo... once that produces a debian-iptables image with modern go-runner images, we can pick that up in 1.23 and 1.24.

/transfer release

[liggitt]

https://github.com/kubernetes/release/blob/master/images/build/debian-iptables/variants.yaml says IMAGE_VERSION: 'bullseye-v1.5.1' is for Kubernetes 1.23 and newer, but 1.23 is still on bullseye-v1.1.0 and 1.24 is still on bullseye-v1.3.0... should those update to 1.5.1+?

[aojea]

I think that this is the change d36a24a

[aojea]

dup of #2507 ?

[liggitt]

d36a24a didn't update the debian-iptables IMAGE_VERSION, and didn't produce a new image, did it? (was 1.5.1 before and is still 1.5.1)

[liggitt]

also, the go-runner image is still stale (should be 1.19.4 now)

[liggitt]

should 1.23 and 1.24 update from bullseye-v1.1.0 and bullseye-v1.3.0 to bullseye-v1.5.1 (or whatever 1.5.x is after we update go-runner 1.19.4)?

[aojea]

d36a24a didn't update the debian-iptables IMAGE_VERSION, and didn't produce a new image, did it? (was 1.5.1 before and is still 1.5.1)

I see new images pushed with the same tag
https://console.cloud.google.com/gcr/images/k8s-staging-build-image/global/debian-iptables?project=k8s-staging-build-image

[aojea]

should 1.23 and 1.24 update from bullseye-v1.1.0 and bullseye-v1.3.0 to bullseye-v1.5.1 (or whatever 1.5.x is after we update go-runner 1.19.4)?

I think so

[liggitt]

I see new images pushed with the same tag
https://console.cloud.google.com/gcr/images/k8s-staging-build-image/global/debian-iptables?project=k8s-staging-build-image

really? 😱 I didn't expect mutable tags there

[BenTheElder]

FWIW we normally don't consume from staging directly and what we promote to production (registry.k8s.io) is promoted by digest so the tag mutability isn't as problematic as it sounds.

[liggitt]

do we know if the 1.5.1 image that got promoted included the go-runner update or not?

[jhawkins1]

@liggitt , in January we updated to 1.24.10, and it is still using the old go-runner. So, it does not look like this has been corrected unless it made it into the 1.24.11 release that just came out -- I could not really tell from the CHANGELOG comments.

[liggitt]

1.23 (EOL):

• https://github.com/kubernetes/kubernetes/blob/release-1.23/build/common.sh#L93
• references debian-iptables v1.1.0

• release/images/build/debian-iptables/Makefile

  Lines 21 to 24 in b874bde

  	IMAGE_VERSION ?= bullseye-v1.1.0
  	CONFIG ?= bullseye
  	DEBIAN_BASE_VERSION ?= bullseye-v1.1.0
  	GORUNNER_VERSION ?= v2.3.1-go1.17.3-bullseye.0

• references gorunner built with go1.17.x

1.24:

• https://github.com/kubernetes/kubernetes/blob/release-1.24/build/common.sh#L93
• references debian-iptables v1.3.0

• release/images/build/debian-iptables/Makefile

  Lines 21 to 24 in 0ced957

  	IMAGE_VERSION ?= bullseye-v1.3.0
  	CONFIG ?= bullseye
  	DEBIAN_BASE_VERSION ?= bullseye-v1.2.0
  	GORUNNER_VERSION ?= v2.3.1-go1.17.3-bullseye.0

• references gorunner built with go1.17.x

1.25:

• https://github.com/kubernetes/kubernetes/blob/release-1.25/build/common.sh#L93
• references distroless-iptables v0.1.0

• release/images/build/distroless-iptables/Makefile

  Lines 21 to 24 in 2c2e73f

  	IMAGE_VERSION ?= v0.1.1
  	CONFIG ?= distroless
  	BASEIMAGE ?= debian:bullseye-slim
  	GORUNNER_VERSION ?= v2.3.1-go1.17.3-bullseye.0

• references gorunner built with go1.17.x

1.26:

• https://github.com/kubernetes/kubernetes/blob/release-1.26/build/common.sh#L98
• references distroless-iptables v0.1.2

• release/images/build/distroless-iptables/Makefile

  Lines 21 to 24 in 44d60ec

  	IMAGE_VERSION ?= v0.1.2
  	CONFIG ?= distroless
  	BASEIMAGE ?= debian:bullseye-slim
  	GORUNNER_VERSION ?= v2.3.1-go1.19.1-bullseye.0

• references gorunner built with go1.19.1 (should update to go1.19.6)

[liggitt]

cc @cpanato for visibility / gap in dependency tracking / propagating new go versions through image chains

[cpanato]

looks like in the past we missed to update distroless-iptables, but now we are tracking that and keeping up-to-date

i will review the active branches and check those things

[cpanato]

@liggitt should we build a new distrolles-iptables using go 1.19 and update that in the active patch branches?

today we are building that only with go 1.20

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.