docs: adding doc for pubsub (#2808)

Signed-off-by: Jaydip Gabani <gabanijaydip@gmail.com> Co-authored-by: Sertaç Özercan <852750+sozercan@users.noreply.github.com>
open-policy-agent · Aug 1, 2023 · 7dda9d2 · 7dda9d2
1 parent 7722c8c
commit 7dda9d2
Show file tree

Hide file tree

Showing 4 changed files with 310 additions and 3 deletions.
diff --git a/website/docs/audit.md b/website/docs/audit.md
@@ -71,6 +71,10 @@ status:
     name: kube-system
 ```
 
+Limitations of getting violations from constraint status:
+
+- To reduce in-memory consumption of Gatekeeper audit pod and to avoid hitting [default etcd limit](https://etcd.io/docs/v3.5/dev-guide/limit/#request-size-limit) of 1.5MB per resource, gatekeeper recommends configuring a [limit up-to 500 violations](https://open-policy-agent.github.io/gatekeeper/website/docs/audit/#configuring-audit)(by default 20) on constraint. Because of these limitations, users might not get all the violations from a Constraint resource.
+
 ### Audit Logs
 
 #### Violations
@@ -113,6 +117,10 @@ audit log entries also contain:
 * An `audit_id` field that uniquely identifies a given audit run. This allows indexing of historical audits
 * An `event_type` field with a value of `violation_audited` to make it easy to programatically identify audit violations
 
+Limitations of getting violations from audit logs:
+
+- It could be difficult to parse audit pod logs to look for violation messages, as violation logs would be mixed together with other log statements.
+
 #### Other Event Types
 
 In addition to violations, these other audit events may be useful (all uniquely identified via the `event_type` field):
@@ -124,13 +132,22 @@ In addition to violations, these other audit events may be useful (all uniquely
 All of these events (including `violation_audited`) are marked 
 with the same `audit_id` for a given audit run.
 
+### Pubsub channel
+
+This feature uses publish and subscribe (pubsub) model that allows Gatekeeper to export audit violations over a broker that can be consumed by a subscriber independently. Therefore, pubsub violations are not subject to reporting limits. Please refer to [this](pubsub.md) guide to configure audit to push violations over a channel.
+
+Limitations/drawbacks of getting violations using pubsub channel:
+
+- There is an inherent risk of messages getting dropped. You might not receive all the published violations.
+- Additional dependancy on pubsub broker. 
+
 ## Running Audit
 For more details on how to deploy audit and 
 number of instances to run, please refer to [operations audit](operations.md#audit). 
 
 ## Configuring Audit
 
-- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`)
+- Audit violations per constraint: set `--constraint-violations-limit=123` (defaults to `20`). NOTE: This flag only impacts when gathering audit results using the constraint status model. If you are gathering audit results using the pubsub model, please refer to the [pubsub](pubsub.md) guide. Both approaches for getting audit violations can be configured independently and work simultaneously without any interference.
 - Audit chunk size: set `--audit-chunk-size=400` (defaults to `500`, `0` = infinite) Lower chunk size can reduce memory consumption of the auditing `Pod` but can increase the number requests to the Kubernetes API server.
 - Audit interval: set `--audit-interval=123` (defaults to every `60` seconds). Disable audit interval by setting `--audit-interval=0`
 - Audit api server cache write to disk (Gatekeeper v3.7.0+): Starting from v3.7.0, by default, audit writes api server cache to the disk attached to the node. This reduces the memory consumption of the audit `pod`. If there are concerns with high IOPS, then switch audit to write cache to a tmpfs ramdisk instead. NOTE: write to ramdisk will increase memory footprint of the audit `pod`.  

diff --git a/website/docs/pubsub-driver-walkthrough.md b/website/docs/pubsub-driver-walkthrough.md
@@ -0,0 +1,61 @@
+---
+id: pubsub-driver
+title: Pubsub Interface/Driver walkthrough
+---
+
+This guide provides an overview of the pubsub interface, including details on its structure and functionality. Additionally, it offers instructions on adding a new driver and utilizing providers other than the default provider Dapr.
+
+## Pubsub interface and Driver walkthrough
+
+Pubsub's connection interface looks like
+```go
+// Connection is the interface that wraps pubsub methods.
+type Connection interface {
+	// Publish single message over a specific topic/channel
+	Publish(ctx context.Context, message interface{}, topic string) error
+
+	// Close connections
+	CloseConnection() error
+
+	// Update an existing connection with new configuration
+	UpdateConnection(ctx context.Context, config interface{}) error
+}
+```
+
+As an example, the Dapr driver implements these three methods to publish message, close connection, and update connection respectively. Please refer to [dapr.go](https://github.com/open-policy-agent/gatekeeper/blob/master/pkg/pubsub/dapr/dapr.go) to understand the logic that goes in each of these methods. Additionally, the Dapr driver also implements `func NewConnection(_ context.Context, config interface{}) (connection.Connection, error)` method that returns a new client for dapr.
+
+### How to add new drivers
+
+**Note:** For example, if we want to add a new driver to use `foo` instead of Dapr as a tool to publish violations.
+
+A driver must implement the `Connection` interface and a new `func NewConnection(_ context.Context, config interface{}) (connection.Connection, error)` method that returns a client for the respective tool.
+
+This newly added driver's `NewConnection` method must be used to create a new `pubSubs` object in [provider.go](https://github.com/open-policy-agent/gatekeeper/blob/master/pkg/pubsub/provider/provider.go). For example,
+
+```go
+var pubSubs = newPubSubSet(map[string]InitiateConnection{
+  dapr.Name: dapr.NewConnection,
+  "foo": foo.NewConnection,
+},
+)
+```
+
+### How to use different  providers
+
+To enable audit to use this driver to publish messages, a connection configMap with appropriate `config` and `provider` is needed. For example,
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: audit
+  namespace: gatekeeper-system
+data:
+  provider: "foo"
+  config: |
+    {
+      <config needed for foo connection>
+    }
+```
+
+> The `data.provider` field must exist and must match one of the keys of the `pubSubs` map that was defined earlier to use the corresponding driver. The `data.config` field in the configuration can vary depending on the driver being used. For dapr driver, `data.config` must be `{"component": "pubsub"}`.
diff --git a/website/docs/pubsub.md b/website/docs/pubsub.md
@@ -0,0 +1,227 @@
+---
+id: pubsub
+title: Consuming violations using Pubsub
+---
+
+`Feature State`: Gatekeeper version v3.13+ (alpha)
+
+> ❗ This feature is alpha, subject to change (feedback is welcome!).
+
+## Description
+
+This feature pushes audit violations to a pubsub service. Users can subscribe to pubsub service to consume violations.
+
+> To gain insights into different methods of obtaining audit violations and the respective trade-offs for each approach, please refer to [Reading Audit Results](audit.md#reading-audit-results).
+
+## Enabling Gatekeeper to export audit violations
+
+Install prerequisites such as a pubsub tool, a message broker etc.
+
+### Setting up audit with pubsub enabled
+
+In the audit deployment, set the `--enable-pub-sub` flag to `true` to publish audit violations. Additionally, `--audit-connection` and `--audit-channel` flags must be set to allow audit to publish violations. `--audit-connection` must be set to the name of the connection config, and `--audit-channel` must be set to name of the channel where violations should get published.
+
+Create a connection configMap that supplies [a provider-specific configuration](pubsub-driver-walkthrough.md#how-to-use-different-providers) for a connection to get established. For instance, to establish a connection that uses Dapr to publish messages this configMap is appropriate:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: audit-pubsub-connection
+  namespace: gatekeeper-system
+data:
+  provider: "dapr"
+  config: |
+    {
+      "component": "pubsub"
+    }
+```
+
+- `provider` field determines which tool/driver should be used to establish a connection. Valid values are: `dapr`
+- `config` field is a json object that configures how the connection is made. E.g. which queue messages should be sent to.
+
+#### Available Pubsub drivers
+Dapr: https://dapr.io/
+
+### Violations
+
+The audit pod publishes violations in following format:
+
+```json
+{
+  "id": "2023-07-18T21:21:52Z",
+  "details": {
+    "missing_labels": [
+      "test"
+    ]
+  },
+  "eventType": "violation_audited",
+  "group": "constraints.gatekeeper.sh",
+  "version": "v1beta1",
+  "kind": "K8sRequiredLabels",
+  "name": "pod-must-have-test",
+  "message": "you must provide labels: {\"test\"}",
+  "enforcementAction": "deny",
+  "resourceAPIVersion": "v1",
+  "resourceKind": "Pod",
+  "resourceNamespace": "nginx",
+  "resourceName": "nginx-deployment-cd55c47f5-2b84x",
+  "resourceLabels": {
+    "app": "nginx",
+    "pod-template-hash": "cd55c47f5"
+  }
+}
+```
+
+### Quick start with publishing violations using Dapr and Redis
+
+> Redis is used for example purposes only. Dapr supports [many different state store options](https://docs.dapr.io/reference/components-reference/supported-state-stores/).
+
+#### Prerequisites
+
+1. Install Dapr
+
+    To install Dapr with specific requirements and configuration, please refer to [Dapr docs](https://docs.dapr.io/operations/hosting/kubernetes/kubernetes-deploy/).
+    > Dapr is installed with mtls enabled by default, for more details on the same plaase refer to [Dapr security](https://docs.dapr.io/operations/security/mtls/#setting-up-mtls-with-the-configuration-resource).
+
+2. Install Redis
+
+    Please refer to [this](https://docs.dapr.io/getting-started/tutorials/configure-state-pubsub/#step-1-create-a-redis-store) guide to install Redis.
+
+    > To install Redis with TLS, please refer to [this](https://docs.bitnami.com/kubernetes/infrastructure/redis-cluster/administration/enable-tls/) doc.
+
+#### Configure a sample subscriber to receive violations
+
+1. Create `fake-subscriber` namespace and redis secret
+
+```shell
+kubectl create ns fake-subscriber
+kubectl get secret redis --namespace=default -o yaml | sed 's/namespace: .*/namespace: fake-subscriber/' | kubectl apply -f - # creating redis secret in subscriber namespace to allow dapr sidecar to connect to redis instance
+```
+
+2. Create Dapr pubsub component
+```shell
+kubectl apply -f <<EOF
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: pubsub
+  namespace: fake-subscriber
+spec:
+  type: pubsub.redis
+  version: v1
+  metadata:
+  - name: redisHost
+    value: redis-master.default.svc.cluster.local:6379
+  - name: redisPassword
+    secretKeyRef: 
+      name: redis
+      key: redis-password
+```
+> Please use [this guide](https://docs.dapr.io/reference/components-reference/supported-state-stores/setup-redis/) to properly configure Redis pubsub component for Dapr.
+
+3. Deploy subscriber application
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sub
+  namespace: fake-subscriber
+  labels:
+    app: sub
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: sub
+  template:
+    metadata:
+      labels:
+        app: sub
+      annotations:
+        dapr.io/enabled: "true"
+        dapr.io/app-id: "subscriber"
+        dapr.io/enable-api-logging: "true"
+        dapr.io/app-port: "6002"
+    spec:
+      containers:
+      - name: go-sub
+        image: fake-subscriber:latest
+        imagePullPolicy: Never
+```
+**Note:** Dockerfile to build image for fake-subscriber is under [gatekeeper/test/fake-subscriber](https://github.com/open-policy-agent/gatekeeper/tree/master/test/pubsub/fake-subscriber). You can find make rule to build and deploy subscriber in [Makefile](https://github.com/open-policy-agent/gatekeeper/blob/master/Makefile) under name `e2e-subscriber-build-load-image` and `e2e-subscriber-deploy`.
+
+#### Configure Gatekeeper with Pubsub enabled
+
+1. Create Dapr pubsub component and Redis secret in Gatekeeper's namespace (`gatekeeper-system` by default). Please make sure to update `gatekeeper-system` namespace for the next steps if your cluster's Gatekeeper namespace is different.
+
+```shell
+kubectl get secret redis --namespace=default -o yaml | sed 's/namespace: .*/namespace: gatekeeper-system/' | kubectl apply -f -
+kubectl apply -f - <<EOF
+apiVersion: dapr.io/v1alpha1
+kind: Component
+metadata:
+  name: pubsub
+  namespace: gatekeeper-system
+spec:
+  type: pubsub.redis
+  version: v1
+  metadata:
+  - name: redisHost
+    value: redis-master.default.svc.cluster.local:6379
+  - name: redisPassword
+    secretKeyRef:
+      name: redis
+      key: redis-password
+EOF
+```
+
+2. Install Gatekeeper with `--enable-pub-sub` set to `true`, `--audit-connection` set to `audit-pubsub-connection`, `--audit-channel` set to `audit` on audit pod.
+
+
+```shell
+echo 'auditPodAnnotations: {dapr.io/enabled: "true", dapr.io/app-id: "audit", dapr.io/metrics-port: "9999"}' > .tmp/annotations.yaml # auditPodAnnotations is used to add annotations required by Dapr to inject sidecar to audit pod
+helm install gatekeeper/gatekeeper --name-template=gatekeeper --namespace gatekeeper-system \
+--set audit.enablePubsub=true \
+--set audit.connection=audit-pubsub-connection \
+--set audit.channel=audit \
+--values .tmp/annotations.yaml
+```
+
+**Note:** Verify that after the audit pod is running there is a dapr sidecar injected and running along side `manager` container.
+
+3. Create connection config to establish a connection.
+
+```shell
+kubectl apply -f - <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: audit-pubsub-connection
+  namespace: gatekeeper-system
+data:
+  provider: "dapr"
+  config: |
+    {
+      "component": "pubsub"
+    }
+EOF
+```
+**Note:** Name of the connection configMap must match the value of `--audit-connection` for it to be used by audit to publish violation. At the moment, now only one connection config can exists for audit.
+
+4. Create the constraint templates and constraints, and make sure audit ran by checking constraints. If constaint status is updated with information such as `auditTimeStamp` or `totalViolations`, then audit has ran atleast once. Additionally, populated `TOTAL-VIOLATIONS` field for all constraints while lising constraints also indicates that audit has ran at least once.
+
+```log
+kubctl get constraint
+NAME                 ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
+pod-must-have-test                        0
+```
+
+
+5. Finally, check the subscriber logs to see the violations received.
+
+```log
+kubectl logs pod/sub-57bd5d694-7lvzf -c go-sub -n fake-subscriber 
+2023/07/18 20:16:41 Listening...
+2023/07/18 20:37:20 main.PubsubMsg{ID:"2023-07-18T20:37:19Z", Details:map[string]interface {}{"missing_labels":[]interface {}{"test"}}, EventType:"violation_audited", Group:"constraints.gatekeeper.sh", Version:"v1beta1", Kind:"K8sRequiredLabels", Name:"pod-must-have-test", Namespace:"", Message:"you must provide labels: {\"test\"}", EnforcementAction:"deny", ConstraintAnnotations:map[string]string(nil), ResourceGroup:"", ResourceAPIVersion:"v1", ResourceKind:"Pod", ResourceNamespace:"nginx", ResourceName:"nginx-deployment-58899467f5-j85bs", ResourceLabels:map[string]string{"app":"nginx", "owner":"admin", "pod-template-hash":"58899467f5"}}
+```
diff --git a/website/sidebars.js b/website/sidebars.js
@@ -33,7 +33,8 @@ module.exports = {
         'externaldata',
         'expansion',
         'gator',
-        'workload-resources'
+        'workload-resources',
+        'pubsub'
       ],
     },
     {
@@ -58,7 +59,8 @@ module.exports = {
       items: [
         'developers',
         'help',
-        'security'
+        'security',
+        'pubsub-driver'
       ],
     }
   ]