Too much memory used with large number of namespaces

[guillaumerose]

Expected Behavior

Same memory usage as the pipeline controller

Actual Behavior

The memory taken by the interceptor and trigger controller is around 1GB.

This is not the case of the pipeline controller which stay around 50MB.

(memory usage of pods in Tekton namespace during the load test - highest are tekton-triggers-controller and tekton-triggers-core-interceptors pods)

Steps to Reproduce the Problem

1. Create 2000 namespaces. In the first 600, create a PipelineRun like this:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: pipeline1
spec:
  pipelineSpec:
    tasks:
      - name: step1
        taskSpec:
          steps:
          - name: step1
            image: busybox
            imagePullPolicy: IfNotPresent
            script: ls /

2. Observe the memory taken by the 2 controllers

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:31:32Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1+d8c4430", GitCommit:"2ade4d46986c94405aa0542f8d6ff7691bddb153", GitTreeState:"clean", BuildDate:"2021-11-02T02:02:53Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

pipeline: v0.28.2
triggers: v0.16.1

• Other additional information

It is stable: if I kill the pod, the controller is recreated and takes almost the same amount of memory.

The cluster has 681 deployments and 14180 configMaps.

I turned on profiling and saw there is something happening in the informers.

[dibyom]

Thanks for the detailed bug report. At a first glance, it looks like the watch for the lister/informer cache may be watching too many things.

[guillaumerose]

I found an explanation for the Triggers controller. See #1273

For the interceptors container:

I observe that all secrets of the cluster are kept in memory. See https://github.com/tektoncd/triggers/blob/main/cmd/interceptors/main.go#L67.
I see 2 possibles solutions:

1. Don't use a secret lister but instead use a secret getter

Pros:

• no more informer
• this is what pipeline already does

Cons:

• for each webhook, the interceptor will issue an API call to get the secret. Perhaps a cache can mitigate this?

2. Ask users to add a specific label on their secrets so that we can filter on them.

Pros:

• less code to change

Cons:

• breaking change for the user.

[guillaumerose]

I also realize that the triggers controller is watching for all deployments and all services in the cluster. It might be a good idea to filter them.

[lbernick]

/priority important-longterm

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[dibyom]

This was mostly fixed. The remaining work is around caching only the secrets needed by the interceptors vs all secrets.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[khrm]

Fixed by #1359