Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.102 cherries #112

Merged
merged 7 commits into from
Oct 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions cmd/gardener-operator/app/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ import (
"github.com/gardener/gardener/pkg/operator/apis/config"
operatorclient "github.com/gardener/gardener/pkg/operator/client"
"github.com/gardener/gardener/pkg/operator/controller"
"github.com/gardener/gardener/pkg/operator/metrics"
"github.com/gardener/gardener/pkg/operator/webhook"
)

Expand Down Expand Up @@ -186,6 +187,11 @@ func run(ctx context.Context, log logr.Logger, cfg *config.OperatorConfiguration
return fmt.Errorf("failed adding webhook handlers to manager: %w", err)
}

log.Info("Adding custom metrics to manager")
if err := metrics.AddToManager(ctx, mgr); err != nil {
return fmt.Errorf("failed adding metrics to manager: %w", err)
}

gardenClientMap, err := clientmapbuilder.
NewGardenClientMapBuilder().
WithRuntimeClient(mgr.GetClient()).
Expand Down
1 change: 1 addition & 0 deletions docs/deployment/feature_gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ A *General Availability* (GA) feature is also referred to as a *stable* feature.
| DefaultSeccompProfile | `gardenlet`, `gardener-operator` | Enables the defaulting of the seccomp profile for Gardener managed workload in the garden or seed to `RuntimeDefault`. |
| IPv6SingleStack | `gardener-apiserver`, `gardenlet` | Allows creating seed and shoot clusters with [IPv6 single-stack networking](../usage/ipv6.md) enabled in their spec ([GEP-21](../proposals/21-ipv6-singlestack-local.md)). If enabled in gardenlet, the default behavior is unchanged, but setting `ipFamilies=[IPv6]` in the `seedConfig` is allowed. Only if the `ipFamilies` setting is changed, gardenlet behaves differently. |
| ShootForceDeletion | `gardener-apiserver` | Allows forceful deletion of Shoots by annotating them with the `confirmation.gardener.cloud/force-deletion` annotation. |
| DisableAPIServerProxyPort | `gardenlet` | Disables the proxy port (8443) on the istio-ingressgateway Services. It was previously used by the apiserver-proxy to route client traffic on the kubernetes Service to the corresponding API server using the TCP proxy protocol. As soon as a shoot has been reconciled by gardener v1.96+, the apiserver-proxy is reconfigured to use HTTP CONNECT on the tls-tunnel port (8132) instead, i.e., it reuses the reversed VPN path to connect to the correct API server. Operators can choose to remove the legacy apiserver-proxy port as soon as all shoots have switched to the new apiserver-proxy configuration. They might want to do so if they activate the ACL extension, which is vulnerable to proxy protocol headers of untrusted clients on the apiserver-proxy port. |
| UseNamespacedCloudProfile | `gardener-apiserver` | Enables usage of `NamespacedCloudProfile`s in `Shoot`s. |
| ShootManagedIssuer | `gardenlet` | Enables the shoot managed issuer functionality described in GEP 24. |
| VPAForETCD | `gardenlet`, `gardener-operator` | Enables VPA for `etcd-main` and `etcd-events`, regardless of HVPA enablement. |
Expand Down
5 changes: 3 additions & 2 deletions imagevector/containers.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -749,8 +749,9 @@ images:
# External Authorization Server for the Istio Endpoint of Reversed VPN
- name: ext-authz-server
sourceRepository: github.com/gardener/ext-authz-server
repository: europe-docker.pkg.dev/gardener-project/releases/gardener/ext-authz-server
tag: "0.10.0"
# built from https://github.com/stackitcloud/ext-authz-server/tree/hackathon-apiserver-proxy
repository: ghcr.io/stackitcloud/ext-authz-server
tag: "0.11.0-dev-818656a"

# API Server SNI
- name: apiserver-proxy
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ ctr images mount "` + image + `" "$tmp_dir"

echo "> Copy gardener-node-agent binary to host (/opt/bin) and make it executable"
mkdir -p "/opt/bin"
cp -f "$tmp_dir/gardener-node-agent" "/opt/bin"
cp -f "$tmp_dir/ko-app/gardener-node-agent" "/opt/bin"
chmod +x "/opt/bin/gardener-node-agent"

echo "> Bootstrap gardener-node-agent"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ ctr images mount "{{ .image }}" "$tmp_dir"

echo "> Copy gardener-node-agent binary to host ({{ .binaryDirectory }}) and make it executable"
mkdir -p "{{ .binaryDirectory }}"
cp -f "$tmp_dir/gardener-node-agent" "{{ .binaryDirectory }}"
cp -f "$tmp_dir/ko-app/gardener-node-agent" "{{ .binaryDirectory }}"
chmod +x "{{ .binaryDirectory }}/gardener-node-agent"

echo "> Bootstrap gardener-node-agent"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ func (component) Config(ctx components.Context) ([]extensionsv1alpha1.Unit, []ex
Content: extensionsv1alpha1.FileContent{
ImageRef: &extensionsv1alpha1.FileContentImageRef{
Image: ctx.Images[imagevector.ContainerImageNameGardenerNodeAgent].String(),
FilePathInImage: "/gardener-node-agent",
FilePathInImage: "/ko-app/gardener-node-agent",
},
},
})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ var _ = Describe("Component", func() {
Content: extensionsv1alpha1.FileContent{
ImageRef: &extensionsv1alpha1.FileContentImageRef{
Image: "gardener-node-agent:v1",
FilePathInImage: "/gardener-node-agent",
FilePathInImage: "/ko-app/gardener-node-agent",
},
},
})
Expand Down
1 change: 1 addition & 0 deletions pkg/component/gardener/apiserver/apiserver_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -584,6 +584,7 @@ var _ = Describe("GardenerAPIServer", func() {
"--log-format=" + logFormat,
"--secure-port=8443",
"--workload-identity-token-issuer=" + workloadIdentityIssuer,
"--shoot-admin-kubeconfig-max-expiration=4320h",
"--workload-identity-signing-key-file=/etc/gardener-apiserver/workload-identity/signing/key.pem",
"--http2-max-streams-per-connection=1000",
"--etcd-cafile=/srv/kubernetes/etcd/ca/bundle.crt",
Expand Down
2 changes: 2 additions & 0 deletions pkg/component/gardener/apiserver/deployment.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ func (g *gardenerAPIServer) deployment(
"--log-format=" + g.values.LogFormat,
fmt.Sprintf("--secure-port=%d", port),
"--workload-identity-token-issuer=" + g.values.WorkloadIdentityTokenIssuer,
// TODO: replace this hardcoded configuration with proper fields in the Garden API
"--shoot-admin-kubeconfig-max-expiration=4320h", // 6 months
},
Ports: []corev1.ContainerPort{{
Name: "https",
Expand Down
2 changes: 2 additions & 0 deletions pkg/component/gardener/controllermanager/configmaps.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ func (g *gardenerControllerManager) configMapControllerManagerConfig() (*corev1.
Project: &controllermanagerv1alpha1.ProjectControllerConfiguration{
ConcurrentSyncs: ptr.To(20),
Quotas: g.values.Quotas,
// TODO: replace this hardcoded configuration with proper fields in the Garden API
StaleExpirationTimeDays: ptr.To(6000),
},
SecretBinding: &controllermanagerv1alpha1.SecretBindingControllerConfiguration{
ConcurrentSyncs: ptr.To(20),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,7 @@ var _ = Describe("GardenerControllerManager", func() {
managedResourceSecretRuntime.Name = managedResourceRuntime.Spec.SecretRefs[0].Name
Expect(fakeClient.Get(ctx, client.ObjectKeyFromObject(managedResourceSecretRuntime), managedResourceSecretRuntime)).To(Succeed())
cm := configMap(namespace, values)
Expect(cm.Name).To(Equal("gardener-controller-manager-config-960e3f19"))
Expect(cm.Name).To(Equal("gardener-controller-manager-config-625036ea"))
expectedRuntimeObjects = []client.Object{
cm,
serviceRuntime,
Expand Down Expand Up @@ -720,8 +720,9 @@ func configMap(namespace string, testValues Values) *corev1.ConfigMap {
ConcurrentSyncs: ptr.To(20),
},
Project: &controllermanagerv1alpha1.ProjectControllerConfiguration{
ConcurrentSyncs: ptr.To(20),
Quotas: testValues.Quotas,
ConcurrentSyncs: ptr.To(20),
Quotas: testValues.Quotas,
StaleExpirationTimeDays: ptr.To(6000),
},
SecretBinding: &controllermanagerv1alpha1.SecretBindingControllerConfiguration{
ConcurrentSyncs: ptr.To(20),
Expand Down
3 changes: 3 additions & 0 deletions pkg/component/gardener/resourcemanager/resource_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -602,6 +602,9 @@ func (r *resourceManager) ensureConfigMap(ctx context.Context, configMap *corev1
}, r.values.NetworkPolicyAdditionalNamespaceSelectors...),
IngressControllerSelector: r.values.NetworkPolicyControllerIngressControllerSelector,
}

config.SourceClientConnection.ClientConnectionConfiguration.QPS = 300
config.SourceClientConnection.ClientConnectionConfiguration.Burst = 500
config.Webhooks.CRDDeletionProtection.Enabled = true
config.Webhooks.ExtensionValidation.Enabled = true
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -497,6 +497,8 @@ var _ = Describe("ResourceManager", func() {
},
IngressControllerSelector: ingressControllerSelector,
}
config.SourceClientConnection.ClientConnectionConfiguration.QPS = 300
config.SourceClientConnection.ClientConnectionConfiguration.Burst = 500
config.Webhooks.CRDDeletionProtection.Enabled = true
config.Webhooks.EndpointSliceHints.Enabled = true
config.Webhooks.ExtensionValidation.Enabled = true
Expand Down
4 changes: 4 additions & 0 deletions pkg/component/kubernetes/apiserverexposure/sni.go
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,10 @@ func (s *sni) Deploy(ctx context.Context) error {
if err := managedresources.CreateForSeed(ctx, s.client, s.namespace, managedResourceName, false, serializedObjects); err != nil {
return err
}
} else {
if err := managedresources.DeleteForSeed(ctx, s.client, s.namespace, managedResourceName); err != nil {
return err
}
}

if _, err := controllerutils.GetAndCreateOrMergePatch(ctx, s.client, destinationRule, istio.DestinationRuleWithLocalityPreference(destinationRule, getLabels(), hostName)); err != nil {
Expand Down
8 changes: 7 additions & 1 deletion pkg/component/kubernetes/apiserverexposure/sni_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ var _ = Describe("#SNI", func() {

if apiServerProxyValues != nil {
managedResource := &resourcesv1alpha1.ManagedResource{}
Expect(c.Get(ctx, client.ObjectKey{Namespace: expectedManagedResource.Namespace, Name: expectedManagedResource.Name}, managedResource)).To(Succeed())
Expect(c.Get(ctx, client.ObjectKeyFromObject(expectedManagedResource), managedResource)).To(Succeed())
expectedManagedResource.Spec.SecretRefs = []corev1.LocalObjectReference{{Name: managedResource.Spec.SecretRefs[0].Name}}
utilruntime.Must(references.InjectAnnotations(expectedManagedResource))
Expect(managedResource).To(DeepEqual(expectedManagedResource))
Expand All @@ -241,6 +241,8 @@ var _ = Describe("#SNI", func() {
actualEnvoyFilter := managedResourceEnvoyFilter.(*istionetworkingv1alpha3.EnvoyFilter)
// cannot validate the Spec as there is no meaningful way to unmarshal the data into the Golang structure
Expect(actualEnvoyFilter.ObjectMeta).To(DeepEqual(expectedEnvoyFilterObjectMeta))
} else {
Expect(c.Get(ctx, client.ObjectKeyFromObject(expectedManagedResource), &resourcesv1alpha1.ManagedResource{})).To(BeNotFoundError(), "should delete ManagedResource for apiserver-proxy EnvoyFilter")
}
}

Expand All @@ -253,6 +255,10 @@ var _ = Describe("#SNI", func() {
Context("when APIServer Proxy is not configured", func() {
BeforeEach(func() {
apiServerProxyValues = nil

// create ManagedResource to ensure that Deploy deletes it
expectedManagedResource.ResourceVersion = ""
Expect(c.Create(ctx, expectedManagedResource)).To(Succeed())
})

It("should succeed deploying", func() {
Expand Down
4 changes: 3 additions & 1 deletion pkg/component/networking/apiserverproxy/apiserver_proxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
v1beta1constants "github.com/gardener/gardener/pkg/apis/core/v1beta1/constants"
"github.com/gardener/gardener/pkg/client/kubernetes"
"github.com/gardener/gardener/pkg/component"
"github.com/gardener/gardener/pkg/component/networking/vpn/seedserver"
"github.com/gardener/gardener/pkg/component/observability/monitoring/prometheus/shoot"
monitoringutils "github.com/gardener/gardener/pkg/component/observability/monitoring/utils"
"github.com/gardener/gardener/pkg/controllerutils"
Expand All @@ -43,7 +44,7 @@ const (
name = "apiserver-proxy"

adminPort = 16910
proxySeedServerPort = 8443
proxySeedServerPort = seedserver.GatewayPort
portNameMetrics = "metrics"

volumeNameConfig = "proxy-config"
Expand Down Expand Up @@ -207,6 +208,7 @@ func (a *apiserverProxy) computeResourcesData() (map[string][]byte, error) {
"adminPort": adminPort,
"proxySeedServerHost": a.values.ProxySeedServerHost,
"proxySeedServerPort": proxySeedServerPort,
"namespace": a.namespace,
}); err != nil {
return nil, err
}
Expand Down
22 changes: 9 additions & 13 deletions pkg/component/networking/apiserverproxy/apiserver_proxy_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -268,7 +268,7 @@ var _ = Describe("APIServerProxy", func() {

Context("IPv4", func() {
It("should deploy the managed resource successfully", func() {
test("607149fb")
test("e2e0f68e")
})
})

Expand All @@ -279,7 +279,7 @@ var _ = Describe("APIServerProxy", func() {
})

It("should deploy the managed resource successfully", func() {
test("3fdb1aaf")
test("5e1c6737")
})
})
})
Expand Down Expand Up @@ -462,6 +462,12 @@ static_resources:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: kube_apiserver
cluster: kube_apiserver
tunneling_config:
hostname: "api.internal.local.:443"
headers_to_add:
- header:
key: Reversed-VPN
value: "outbound|443||kube-apiserver.some-namespace.svc.cluster.local"
access_log:
- name: envoy.access_loggers.stdout
typed_config:
Expand Down Expand Up @@ -534,17 +540,7 @@ static_resources:
address:
socket_address:
address: api.internal.local.
port_value: 8443
transport_socket:
name: envoy.transport_sockets.upstream_proxy_protocol
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.proxy_protocol.v3.ProxyProtocolUpstreamTransport
config:
version: V2
transport_socket:
name: envoy.transport_sockets.raw_buffer
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer
port_value: 8132
upstream_connection_options:
tcp_keepalive:
keepalive_time: 7200
Expand Down
16 changes: 6 additions & 10 deletions pkg/component/networking/apiserverproxy/templates/envoy.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,12 @@ static_resources:
"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
stat_prefix: kube_apiserver
cluster: kube_apiserver
tunneling_config:
hostname: "{{ .proxySeedServerHost }}:443"
headers_to_add:
- header:
key: Reversed-VPN
value: "outbound|443||kube-apiserver.{{ .namespace }}.svc.cluster.local"
access_log:
- name: envoy.access_loggers.stdout
typed_config:
Expand Down Expand Up @@ -124,16 +130,6 @@ static_resources:
socket_address:
address: {{ .proxySeedServerHost }}
port_value: {{ .proxySeedServerPort }}
transport_socket:
name: envoy.transport_sockets.upstream_proxy_protocol
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.proxy_protocol.v3.ProxyProtocolUpstreamTransport
config:
version: V2
transport_socket:
name: envoy.transport_sockets.raw_buffer
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer
upstream_connection_options:
tcp_keepalive:
keepalive_time: 7200
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ metrics_path: /metrics/cadvisor

tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: {{.IsManagedSeed}}
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ scheme: https

tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: {{.IsManagedSeed}}
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ metrics_path: /metrics/cadvisor

tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:
Expand Down Expand Up @@ -183,7 +183,7 @@ scheme: https

tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

kubernetes_sd_configs:
Expand Down
1 change: 1 addition & 0 deletions pkg/component/shared/resourcemanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ func NewRuntimeGardenerResourceManager(

return resourcemanager.New(c, gardenNamespaceName, secretsManager, resourcemanager.Values{
ConcurrentSyncs: ptr.To(20),
AlwaysUpdate: ptr.To(true),
DefaultSeccompProfileEnabled: defaultSeccompProfileEnabled,
DefaultNotReadyToleration: defaultNotReadyToleration,
DefaultUnreachableToleration: defaultUnreachableToleration,
Expand Down
Loading