Not able to see violations in constraints. #1783

bj-1795 · 2022-01-07T08:50:52Z

I have installed op-gatekeeper by using following values:

auditInterval: 3600
auditMatchKindOnly: true
constraintViolationsLimit: 20
auditFromCache: true
disableValidatingWebhook: false
validatingWebhookTimeoutSeconds: 3
validatingWebhookFailurePolicy: Ignore
validatingWebhookCheckIgnoreFailurePolicy: Fail
enableDeleteOperations: false
experimentalEnableMutation: false
mutatingWebhookFailurePolicy: Ignore
mutatingWebhookTimeoutSeconds: 3
auditChunkSize: 500
logLevel: WARNING
logDenies: false
emitAdmissionEvents: false
emitAuditEvents: false
resourceQuota: true
postInstall:
  labelNamespace:
    enabled: true
    image:
      repository: openpolicyagent/gatekeeper-crds
      tag: v3.7.0-beta.2
      pullPolicy: IfNotPresent
      pullSecrets: []
image:
  repository: openpolicyagent/gatekeeper
  crdRepository: openpolicyagent/gatekeeper-crds
  release: v3.7.0-beta.2
  pullPolicy: IfNotPresent
  pullSecrets: []
podAnnotations:
  { container.seccomp.security.alpha.kubernetes.io/manager: runtime/default }
podLabels: {}
podCountLimit: 100
secretAnnotations: {}
controllerManager:
  exemptNamespaces: []
  exemptNamespacePrefixes: []
  hostNetwork: false
  port: 8443
  metricsPort: 8888
  healthPort: 9090
  priorityClassName: system-cluster-critical
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: gatekeeper.sh/operation
                  operator: In
                  values:
                    - webhook
            topologyKey: kubernetes.io/hostname
          weight: 100
  tolerations: []
  nodeSelector: { kubernetes.io/os: linux }
  resources:
    limits:
      cpu: 1000m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
audit:
  hostNetwork: false
  metricsPort: 8888
  healthPort: 9090
  priorityClassName: system-cluster-critical
  affinity: {}
  tolerations: []
  nodeSelector: { kubernetes.io/os: linux }
  resources:
    limits:
      cpu: 1000m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
pdb:
  controllerManager:
    minAvailable: 1
service: {}
disabledBuiltins:
psp:
  enabled: true
upgradeCRDs:
  enabled: true

We have a huge cluster and multiple constraints for different namespaces.
I getting Total Violations: 0 which is not correct , in logs getting below error:

{"level":"error","ts":1641387964.9383755,"logger":"controller-runtime.manager.controller.constrainttemplate-controller","msg":"Reconciler error","name":"k8sallowedrepos","namespace":"","error":"customresourcedefinitions.apiextensions.k8s.io \"k8sallowedrepos.constraints.gatekeeper.sh\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"} ```

The text was updated successfully, but these errors were encountered:

maxsmythe · 2022-01-08T00:35:18Z

Just to be sure: audit only reports violations by resources that already exist on the cluster, it doesn't report requests that are denied by the admission webhook. You were expecting to find pre-existing violations, correct?

The log error is likely a red herring -- that's a transient error due to more than one pod attempting to create a constraint template CRD and shouldn't affect audit.

Can you post the contents of a constraint where you expect to see violations but aren't seeing any, particularly the status section? Posting the associated constraint template an a pre-existing resource that should be raising a violation would also be helpful.

Also, looking through any log entries on the audit pod that have the term "process":"audit" could be helpful, as any audit-specific logs should have that tag.

bj-1795 · 2022-01-10T05:24:27Z

pod/gatekeeper-audit-79f88d4655-n4qql                1/1     Running   0          11h
pod/gatekeeper-controller-manager-6cdbd7fbcc-f9zht   1/1     Running   0          11h
pod/gatekeeper-controller-manager-6cdbd7fbcc-ts85l   1/1     Running   0          153m
pod/gatekeeper-controller-manager-6cdbd7fbcc-zmp5p   1/1     Running   0          2d16h

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/gatekeeper-webhook-service   ClusterIP   172.20.112.54   <none>        443/TCP   4d16h

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gatekeeper-audit                1/1     1            1           4d16h
deployment.apps/gatekeeper-controller-manager   3/3     3            3           4d16h

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/gatekeeper-audit-79f88d4655                1         1         1       4d16h
replicaset.apps/gatekeeper-controller-manager-6cdbd7fbcc   3         3         3       4d16h

the pods are restarting againg and again and i have given loglevel as error so now not able to see any auditlogs.

Name:         repo-is-openpolicyagent
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  constraints.gatekeeper.sh/v1beta1
Kind:         K8sAllowedRepos
Metadata:
  Creation Timestamp:  2022-01-07T07:18:16Z
  Generation:          1
  Managed Fields:
    API Version:  constraints.gatekeeper.sh/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:enforcementAction:
        f:match:
          .:
          f:kinds:
        f:parameters:
          .:
          f:repos:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-01-07T07:18:16Z
    API Version:  constraints.gatekeeper.sh/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
    Manager:         gatekeeper
    Operation:       Update
    Time:            2022-01-07T07:18:17Z
  Resource Version:  115472838
  UID:               e33efe44-b973-48ff-9d76-8428ce647983
Spec:
  Enforcement Action:  warn
  Match:
    Kinds:
      API Groups:
        apps
      Kinds:
        Deployment
        StatefulSet
        ReplicaSet
  Parameters:
    Repos:
      405524983081.dkr.ecr.ap-southeast-1.amazonaws.com
      476109445501.dkr.ecr.ap-southeast-1.amazonaws.com
      549918006255.dkr.ecr.ap-southeast-1.amazonaws.com
      298863071907.dkr.ecr.ap-southeast-1.amazonaws.com
      476109445501.dkr.ecr.ap-southeast-1.amazonaws.com
      082372545318.dkr.ecr.ap-southeast-1.amazonaws.com
Status:
  Audit Timestamp:  2022-01-10T08:07:47Z
  By Pod:
    Constraint UID:       e33efe44-b973-48ff-9d76-8428ce647983
    Enforced:             true
    Id:                   gatekeeper-audit-79f88d4655-n4qql
    Observed Generation:  1
    Operations:
      audit
      status
    Constraint UID:       e33efe44-b973-48ff-9d76-8428ce647983
    Enforced:             true
    Id:                   gatekeeper-controller-manager-6cdbd7fbcc-85w9l
    Observed Generation:  1
    Operations:
      webhook
    Constraint UID:       e33efe44-b973-48ff-9d76-8428ce647983
    Enforced:             true
    Id:                   gatekeeper-controller-manager-6cdbd7fbcc-f9zht
    Observed Generation:  1
    Operations:
      webhook
    Constraint UID:       e33efe44-b973-48ff-9d76-8428ce647983
    Enforced:             true
    Id:                   gatekeeper-controller-manager-6cdbd7fbcc-ts85l
    Observed Generation:  1
    Operations:
      webhook
  Total Violations:  0
Events:              <none>

maxsmythe · 2022-01-12T02:14:21Z

Ah, that constraint template only checks Pods, not generator resources (Deployments, StatefulSets, etc.), removing spec.match.kinds (or adding pods to it) should fix the issue.

bj-1795 · 2022-01-17T11:15:56Z

I gave --audit-match-kind-only=false but still Total Violations: 0.

@maxsmythe Can you please suggest something else

maxsmythe · 2022-01-19T02:48:47Z

Sorry, that wasn't what I was suggesting.

In the contents of your constraint yaml, delete spec.match.kinds

bj-1795 · 2022-01-19T06:19:34Z

kind: K8sAllowedRepos
metadata:
  name: repo-not-valid
spec:
  enforcementAction: warn
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment","StatefulSet","ReplicaSet"]
  parameters:
    repos:
      - "405524983081.dkr.ecr.ap-southeast-1.amazonaws.com"
      - "476109445501.dkr.ecr.ap-southeast-1.amazonaws.com"
      - "549918006255.dkr.ecr.ap-southeast-1.amazonaws.com"
      - "298863071907.dkr.ecr.ap-southeast-1.amazonaws.com"
      - "476109445501.dkr.ecr.ap-southeast-1.amazonaws.com"
      - "082372545318.dkr.ecr.ap-southeast-1.amazonaws.com"

this is my constraint template i want to just see violation for kinds : Deployment","StatefulSet","ReplicaSet, i am able to see voilations for my sandbox account not for my nonprod (which is a heavy cluster with lots of deployments), i have changed auditinterval and chunksize as i was facing OOM killed issues, could that be a reason?

maxsmythe · 2022-01-20T01:02:32Z

I'm curious what violations you're seeing? Throwing violations for Deployments, etc. that create invalid pods is not something Gatekeeper currently does.

Fixes open-policy-agent#1783 Signed-off-by: Max Smythe <smythe@google.com>

Fixes #1783 Signed-off-by: Max Smythe <smythe@google.com>

maxsmythe · 2022-05-04T23:03:43Z

Found the core problem -- this bit of Rego code below was ill-behaved when data.external is undefined, which can happen when modifying the config resource. Fixes (attached to this bug) are rolling out.

https://github.com/open-policy-agent/frameworks/blob/8066162cf5e2ba18308c339e2ab57f834948db22/constraint/pkg/client/drivers/local/rego.go#L37

…agent#2038) Fixes open-policy-agent#1783 Signed-off-by: Max Smythe <smythe@google.com> Signed-off-by: davis-haba <davishaba@google.com>

bj-1795 added the bug Something isn't working label Jan 7, 2022

maxsmythe added a commit to maxsmythe/gatekeeper that referenced this issue May 4, 2022

Update CF to fix unenforced violations on data deletion

dfadf17

Fixes open-policy-agent#1783 Signed-off-by: Max Smythe <smythe@google.com>

maxsmythe mentioned this issue May 4, 2022

Update CF to fix unenforced violations on data deletion #2038

Merged

maxsmythe closed this as completed in #2038 May 4, 2022

maxsmythe added a commit that referenced this issue May 4, 2022

Update CF to fix unenforced violations on data deletion (#2038)

e835348

Fixes #1783 Signed-off-by: Max Smythe <smythe@google.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to see violations in constraints. #1783

Not able to see violations in constraints. #1783

bj-1795 commented Jan 7, 2022

maxsmythe commented Jan 8, 2022

bj-1795 commented Jan 10, 2022 •

edited

Loading

maxsmythe commented Jan 12, 2022

bj-1795 commented Jan 17, 2022 •

edited

Loading

maxsmythe commented Jan 19, 2022

bj-1795 commented Jan 19, 2022 •

edited

Loading

maxsmythe commented Jan 20, 2022

maxsmythe commented May 4, 2022

Not able to see violations in constraints. #1783

Not able to see violations in constraints. #1783

Comments

bj-1795 commented Jan 7, 2022

maxsmythe commented Jan 8, 2022

bj-1795 commented Jan 10, 2022 • edited Loading

maxsmythe commented Jan 12, 2022

bj-1795 commented Jan 17, 2022 • edited Loading

maxsmythe commented Jan 19, 2022

bj-1795 commented Jan 19, 2022 • edited Loading

maxsmythe commented Jan 20, 2022

maxsmythe commented May 4, 2022

bj-1795 commented Jan 10, 2022 •

edited

Loading

bj-1795 commented Jan 17, 2022 •

edited

Loading

bj-1795 commented Jan 19, 2022 •

edited

Loading