From 5c514c460569fbeef3cb33a9ec26f1823426db46 Mon Sep 17 00:00:00 2001 From: Rafael Roquetto Date: Mon, 27 Jan 2025 17:42:48 -0600 Subject: [PATCH 1/7] Introduce documentation about required capabilities --- docs/sources/_index.md | 3 +- docs/sources/security.md | 172 +++++++++++++++++++++++++++++++ docs/sources/setup/kubernetes.md | 19 +--- 3 files changed, 176 insertions(+), 18 deletions(-) create mode 100644 docs/sources/security.md diff --git a/docs/sources/_index.md b/docs/sources/_index.md index 91bdc5b82..aa27bca60 100644 --- a/docs/sources/_index.md +++ b/docs/sources/_index.md @@ -73,8 +73,7 @@ While most eBPF programs require elevated privileges, Beyla allow you to specify Some Beyla functionality requires further permissions, for example using the network observability probes with Linux Traffic Control requires `CAP_NET_ADMIN`, but it's a feature you have to optionally enable. -`CAP_SYS_ADMIN` is only required to enable context propagation in Go across multiple nodes, however if this permission -isn't granted Beyla gracefully degrades its functionality to support only partial traces. +For a comprehensive list of capabilities required by Beyla, refer to [Security, permissions and capabilities]({{< relref "./security" >}}). ## Get started diff --git a/docs/sources/security.md b/docs/sources/security.md new file mode 100644 index 000000000..8dd79a80e --- /dev/null +++ b/docs/sources/security.md @@ -0,0 +1,172 @@ +--- +title: Security, permissions and capabilities +menuTitle: Security, permissions and capabilities +description: Privileges and capabilities required by Beyla +weight: 22 +keywords: + - Beyla + - eBPF + - security + - capabilities +aliases: + - /docs/grafana-cloud/monitor-applications/beyla/security/ +--- + +# Security, permissions and capabilities + +Beyla needs access to various Linux interfaces to instrument applications, such as reading from the `/proc` filesystem, loading _eBPF_ programs, and managing network interface filters. Many of these operations require elevated permissions due to their nature. The simplest solution is to run Beyla as `root`, which gives it all the permissions it needs. However, this might not work well in more complex setups where full `root` access isn’t ideal. To address this, Beyla is designed to use only the specific Linux kernel capabilities needed for its current configuration. + +## Linux kernel capabilities + +Linux kernel capabilities are a fine-grained system for controlling access to privileged operations on the system. They allow you to grant specific permissions to processes without giving them full root (superuser) access, which helps improve security by adhering to the principle of least privilege. Instead of relying on the traditional all-or-nothing root model, capabilities split the powerful privileges typically associated with root into smaller, more manageable units. + +Each capability corresponds to a particular privileged operation in the kernel. + +Capabilities are assigned to processes and executable files. By using tools like `setcap`, administrators can assign specific capabilities to a binary, enabling it to perform only the operations it needs without running as `root`. For example: + +```bash +sudo setcap cap_net_admin,cap_net_raw+ep myprogram +``` + +This example grants the `CAP_NET_ADMIN` and `CAP_NET_RAW` capabilities to `myprogram`, allowing it to manage network settings without requiring full superuser privileges. + +By choosing and assigning capabilities carefully, you can lower the risk of privilege escalation while still letting processes do what they need to. + +## Beyla operation modes + +Beyla can operate in two distinct modes: _application observability_ and _network observability_. These modes are not mutually exclusive and can be used together as needed. The former is typically enabled using the `BEYLA_OPEN_PORT` or `BEYLA_EXECUTABLE_NAME` options, while the latter is controlled via the `BEYLA_NETWORK_METRICS` option. For more details on enabling these modes, refer to the [configuration documentation](/docs/beyla/latest/configure/options/). + +What Beyla requires in terms of capabilities depends entirely on the modes and features you enable. Conversely, the capabilities you provide determine what Beyla is able to do. + +When starting up, Beyla reads its configuration and checks for the required capabilities. If any are missing, Beyla displays a warning like the following: + +``` +time=2025-01-27T17:21:20.197-06:00 level=WARN msg="Required system capabilities not present, Beyla may malfunction" error="the following capabilities are required: CAP_DAC_READ_SEARCH, CAP_BPF, CAP_CHECKPOINT_RESTORE" +``` + +Beyla then attempts to continue running, but missing capabilities may lead to errors later on. + +To prevent this, you can set `BEYLA_ENFORCE_SYS_CAPS=1`, which causes Beyla to fail immediately if the required capabilities are not available. In this case, it terminates right after printing the warning message above. + +## List of capabilities required by Beyla + +Below is a list of capabilities and their usage in the context of Beyla + +| Capability | Usage in Beyla | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | +| `CAP_BPF` | enables general BPF functionality and `BPF_PROG_TYPE_SOCK_FILTER` programs | +| `CAP_NET_RAW` | used to create `AF_PACKET` raw sockets | +| `CAP_NET_ADMIN` | required to load `BPF_PROG_TYPE_SCHED_CLS` (tc) programs | +| `CAP_PERFMON` | direct packet access and pointer arithmetic and loading `BPF_PROG_TYPE_KPROBE` programs | +| `CAP_DAC_READ_SEARCH` | access to `/proc/self/mem` to determine kernel version | +| `CAP_CHECKPOINT_RESTORE` | access to symlinks in the `/proc` filesystem | +| `CAP_SYS_PTRACE` | access to `/proc/pid/exe` and friends | +| `CAP_SYS_RESOURCE` | _(kernels **< 5.11** only)_ increase the amount of locked memory available. | +| `CAP_SYS_ADMIN` | Library-level Go trace-context propagation via `bpf_probe_write_user()` and access to BTF data by the BPF metrics exporter | + +**Note** Access to `CAP_PERFMON` is subject to `perf_events` access controls governed by the `kernel.perf_event_paranoid` kernel setting, which can adjusted via `sysctl` or by modifying the file `/proc/sys/kernel/perf_event_paranoid`. +The default setting for `kernel.perf_event_paranoid` is typically `2`, which is documented under the `perf_event_paranoid` section in the [kernel documentation](https://www.kernel.org/doc/Documentation/sysctl/kernel.txt) and more comprehensively under [the perf-security documentation](https://www.kernel.org/doc/Documentation/admin-guide/perf-security.rst). +Some Linux distributions define higher levels for `kernel.perf_event_paranoid`, for example Debian based distributions [also use](https://lwn.net/Articles/696216/) `kernel.perf_event_paranoid=3`, +which disallows access to `perf_event_open()` without `CAP_SYS_ADMIN`. If you are running on a distribution with `kernel.perf_event_paranoid` setting higher than `2`, +you can either modify your configuration to lower it to `2` or use `CAP_SYS_ADMIN` instead of `CAP_PERFMON`. + +## Examples + +The final set of required capabilities depends on the actual Beyla configuration and the type of tracers being used, as described in the section __Beyla operation modes__ above. Here are a few examples of how to run Beyla as a non-root user. + +### Network Metrics (using a socket filter) + +#### Required capabilities: +* `CAP_BPF` +* `CAP_NET_RAW` + +``` +sudo setcap cap_bpf,cap_net_raw+ep ./bin/beyla +BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 bin/beyla +``` + +### Network Metrics (using traffic control) + +#### Required capabilities: +* `CAP_BPF` +* `CAP_NET_ADMIN` +* `CAP_PERFMON` + +``` +sudo setcap cap_bpf,cap_net_admin,cap_perfmon+ep ./bin/beyla +BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 BEYLA_NETWORK_SOURCE=tc bin/beyla +``` + +### Application observability + +#### Required capabilities: +* `CAP_BPF` +* `CAP_DAC_READ_SEARCH` +* `CAP_CHECKPOINT_RESTORE` +* `CAP_PERFMON` +* `CAP_NET_RAW` +* `CAP_SYS_PTRACE` + +``` +sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace+ep ./bin/beyla +BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla +``` + +### Application observability with trace context propagation + +#### Required capabilities: +* `CAP_BPF` +* `CAP_DAC_READ_SEARCH` +* `CAP_CHECKPOINT_RESTORE` +* `CAP_PERFMON` +* `CAP_NET_RAW` +* `CAP_SYS_PTRACE` +* `CAP_NET_ADMIN` + +``` +sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace,cap_net_admin+ep ./bin/beyla +BEYLA_BPF_ENABLE_CONTEXT_PROPAGATION=1 BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla +``` + +## Internal eBPF tracer capability requirement reference + +Below is a list of internal eBPF tracers used by Beyla and their required capabilities +### Socket flow fetcher +- `CAP_BPF` -> for `BPF_PROG_TYPE_SOCK_FILTER` +- `CAP_NET_RAW` -> for creating `AF_PACKET` socket + +### Flow fetcher (tc) +- `CAP_BPF` +- `CAP_NET_ADMIN` -> for `PROG_TYPE_SCHED_CLS` +- `CAP_PERFMON` -> direct access to `struct __sk_buff::data` and pointer arithmetic + +### Watcher +- `CAP_BPF` +- `CAP_CHECKPOINT_RESTORE` +- `CAP_DAC_READ_SEARCH` -> access to `/proc/self/mem` to determine kernel version +- `CAP_PERFMON` -> for `BPF_PROG_TYPE_KPROBE` + +### Generic tracer +- `CAP_BPF` +- `CAP_DAC_READ_SEARCH` +- `CAP_CHECKPOINT_RESTORE` +- `CAP_PERFMON` +- `CAP_NET_RAW` -> for creating `AF_PACKET` socket used by `beyla_socket__http_filter` +- `CAP_SYS_PTRACE` -> access to `/proc/pid/exe` and other nodes in `/proc` + +### TC tracers +* `CAP_BPF` +* `CAP_DAC_READ_SEARCH` +* `CAP_PERFMON` +* `CAP_NET_ADMIN` -> for `BPF_PROG_TYPE_SCHED_CLS`, `BPF_PROG_TYPE_SOCK_OPS` and `BPF_PROG_TYPE_SK_MSG` + +### GO tracer +- `CAP_BPF` +- `CAP_DAC_READ_SEARCH` +- `CAP_CHECKPOINT_RESTORE` +- `CAP_PERFMON` +- `CAP_NET_RAW` -> for creating `AF_PACKET` socket used by `beyla_socket__http_filter` +- `CAP_SYS_PTRACE` -> access to `/proc/pid/exe` and other nodes in `/proc` +- `CAP_SYS_ADMIN` -> for probe based (`bpf_probe_write_user()`) library level context propagation + + diff --git a/docs/sources/setup/kubernetes.md b/docs/sources/setup/kubernetes.md index 11bb8c853..a248c1121 100644 --- a/docs/sources/setup/kubernetes.md +++ b/docs/sources/setup/kubernetes.md @@ -251,24 +251,11 @@ In all of the examples so far, `privileged:true` or the `SYS_ADMIN` Linux capabi The following guide is based on tests performed mainly by running `containerd` with `GKE`, `kubeadm`, `k3s`, `microk8s` and `kind`. -To run Beyla unprivileged, you need to run a `privileged` init container which performs setup tasks which require elevated privileges. Then you need to replace the `privileged:true` setting with a set of Linux [capabilities](https://www.man7.org/linux/man-pages/man7/capabilities.7.html). +To run Beyla unprivileged, you need to run a `privileged` init container which performs setup tasks which require elevated privileges. Then you need to replace the `privileged:true` setting with a set of Linux [capabilities](https://www.man7.org/linux/man-pages/man7/capabilities.7.html). A comprehensive list of capabilities required by Beyla can be found in [Security, permissions and capabilities]({{< relref "../security" >}}). -- `CAP_BPF` is required to install most of the eBPF probes, because Beyla tracks system calls. -- `CAP_SYS_PTRACE` is required so that Beyla is able to look into the processes namespaces and inspect the executables. Beyla doesn't use `ptrace`, but for some of the operations it does require this capability. -- `CAP_NET_RAW` is required for using installing socket filters, which are used as a fallback for `kretprobes` for HTTP requests. -- `CAP_CHECKPOINT_RESTORE` is required to open ELF files. -- `CAP_DAC_READ_SEARCH` is required to open ELF files. -- `CAP_PERFMON` is required to load BPF programs, i.e. be able to perform `perf_event_open()`. -- `CAP_SYS_RESOURCE` is required only on kernels **< 5.11** so that Beyla can increase the amount of locked memory available. +In addition to these Linux capabilities, many Kubernetes versions include [AppArmour](https://kubernetes.io/docs/tutorials/security/apparmor/), which tough policies adds additional restrictions to unprivileged containers. By [default](https://github.com/moby/moby/blob/master/profiles/apparmor/template.go), the AppArmour policy restricts the use of `mount` and the access to `/sys/fs/` directories. Beyla requires access to `/sys/fs/cgroup/`. -In addition to these Linux capabilities, many Kubernetes versions include [AppArmour](https://kubernetes.io/docs/tutorials/security/apparmor/), which tough policies adds additional restrictions to unprivileged containers. By [default](https://github.com/moby/moby/blob/master/profiles/apparmor/template.go), the AppArmour policy restricts the use of `mount` and the access to `/sys/fs/` directories. Beyla uses the BPF Linux file system to store pinned BPF maps, for communication among the different BPF programs. For this reason, Beyla either needs to `mount` a BPF file system, or write to `/sys/fs/bpf`, which are both restricted. - -Because of the AppArmour restriction, to run Beyla as unprivileged container, you need to either: - -- Set `container.apparmor.security.beta.kubernetes.io/beyla: "unconfined"` in your Kubernetes deployment files. -- Set a modified AppArmour policy which allows Beyla to perform `mount`. - -**Note** Since the `beyla` container does not have the privileges required to mount or un-mount the BPF filesystem, this sample leaves the BPF filesystem mounted on the host, even after the sample is deleted. This samples uses a unique path for each namespace to ensure re-use the same mount if Beyla is re-deployed, but to avoid collisions if multiple instances of Beyla is run in different namespaces. +Because of the AppArmour restriction, to run Beyla as unprivileged container, you may need to set `container.apparmor.security.beta.kubernetes.io/beyla: "unconfined"` in your Kubernetes deployment files in case you run into issues. **Note** Loading BPF programs requires that Beyla is able to read the Linux performance events, or at least be able to execute the Linux Kernel API `perf_event_open()`. From 4b1ed77b785bd4ad0998c9fe14d3070d72207145 Mon Sep 17 00:00:00 2001 From: Sean Packham Date: Mon, 3 Feb 2025 14:32:54 +0000 Subject: [PATCH 2/7] Edits --- docs/sources/security.md | 165 +++++++++++++++++++++------------------ 1 file changed, 89 insertions(+), 76 deletions(-) diff --git a/docs/sources/security.md b/docs/sources/security.md index 8dd79a80e..c0be69578 100644 --- a/docs/sources/security.md +++ b/docs/sources/security.md @@ -1,6 +1,6 @@ --- -title: Security, permissions and capabilities -menuTitle: Security, permissions and capabilities +title: Beyla security, permissions, and capabilities +menuTitle: Security description: Privileges and capabilities required by Beyla weight: 22 keywords: @@ -12,17 +12,15 @@ aliases: - /docs/grafana-cloud/monitor-applications/beyla/security/ --- -# Security, permissions and capabilities +# Beyla security, permissions, and capabilities -Beyla needs access to various Linux interfaces to instrument applications, such as reading from the `/proc` filesystem, loading _eBPF_ programs, and managing network interface filters. Many of these operations require elevated permissions due to their nature. The simplest solution is to run Beyla as `root`, which gives it all the permissions it needs. However, this might not work well in more complex setups where full `root` access isn’t ideal. To address this, Beyla is designed to use only the specific Linux kernel capabilities needed for its current configuration. +Beyla needs access to various Linux interfaces to instrument applications, such as reading from the `/proc` filesystem, loading eBPF programs, and managing network interface filters. Many of these operations require elevated permissions. The simplest solution is to run Beyla as `root`, however this might not work well in setups where full `root` access isn’t ideal. To address this, Beyla is designed to use only the specific Linux kernel capabilities needed for its current configuration. ## Linux kernel capabilities -Linux kernel capabilities are a fine-grained system for controlling access to privileged operations on the system. They allow you to grant specific permissions to processes without giving them full root (superuser) access, which helps improve security by adhering to the principle of least privilege. Instead of relying on the traditional all-or-nothing root model, capabilities split the powerful privileges typically associated with root into smaller, more manageable units. +Linux kernel capabilities are a fine-grained system for controlling access to privileged operations. They allow you to grant specific permissions to processes without giving them full superuser or root access, which helps improve security by adhering to the principle of least privilege. Capabilities split privileges typically associated with root into smaller privileged operations in the kernel. -Each capability corresponds to a particular privileged operation in the kernel. - -Capabilities are assigned to processes and executable files. By using tools like `setcap`, administrators can assign specific capabilities to a binary, enabling it to perform only the operations it needs without running as `root`. For example: +Capabilities are assigned to processes and executable files. By using tools like `setcap`, administrators can assign specific capabilities to a binary, enabling it to perform only the operations it needs without running as root. For example: ```bash sudo setcap cap_net_admin,cap_net_raw+ep myprogram @@ -30,15 +28,13 @@ sudo setcap cap_net_admin,cap_net_raw+ep myprogram This example grants the `CAP_NET_ADMIN` and `CAP_NET_RAW` capabilities to `myprogram`, allowing it to manage network settings without requiring full superuser privileges. -By choosing and assigning capabilities carefully, you can lower the risk of privilege escalation while still letting processes do what they need to. +By carefully choosing and assigning capabilities you can lower the risk of privilege escalation while still letting processes do what they need to. ## Beyla operation modes -Beyla can operate in two distinct modes: _application observability_ and _network observability_. These modes are not mutually exclusive and can be used together as needed. The former is typically enabled using the `BEYLA_OPEN_PORT` or `BEYLA_EXECUTABLE_NAME` options, while the latter is controlled via the `BEYLA_NETWORK_METRICS` option. For more details on enabling these modes, refer to the [configuration documentation](/docs/beyla/latest/configure/options/). - -What Beyla requires in terms of capabilities depends entirely on the modes and features you enable. Conversely, the capabilities you provide determine what Beyla is able to do. +Beyla can operate in two distinct modes: *application observability* and *network observability*. These modes are not mutually exclusive and can be used together as needed. For more details on enabling these modes, refer to the [configuration documentation](/docs/beyla/latest/configure/options/). -When starting up, Beyla reads its configuration and checks for the required capabilities. If any are missing, Beyla displays a warning like the following: +Beyla reads its configuration and checks for the required capabilities, if any are missing it displays a warning, for example: ``` time=2025-01-27T17:21:20.197-06:00 level=WARN msg="Required system capabilities not present, Beyla may malfunction" error="the following capabilities are required: CAP_DAC_READ_SEARCH, CAP_BPF, CAP_CHECKPOINT_RESTORE" @@ -46,60 +42,67 @@ time=2025-01-27T17:21:20.197-06:00 level=WARN msg="Required system capabilities Beyla then attempts to continue running, but missing capabilities may lead to errors later on. -To prevent this, you can set `BEYLA_ENFORCE_SYS_CAPS=1`, which causes Beyla to fail immediately if the required capabilities are not available. In this case, it terminates right after printing the warning message above. +You can set `BEYLA_ENFORCE_SYS_CAPS=1`, which causes Beyla to fail immediately if the required capabilities are not available. ## List of capabilities required by Beyla -Below is a list of capabilities and their usage in the context of Beyla +Beyla requires the following The following table Below is a list of capabilities and their usage in the context of Beyla | Capability | Usage in Beyla | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | -| `CAP_BPF` | enables general BPF functionality and `BPF_PROG_TYPE_SOCK_FILTER` programs | -| `CAP_NET_RAW` | used to create `AF_PACKET` raw sockets | -| `CAP_NET_ADMIN` | required to load `BPF_PROG_TYPE_SCHED_CLS` (tc) programs | -| `CAP_PERFMON` | direct packet access and pointer arithmetic and loading `BPF_PROG_TYPE_KPROBE` programs | -| `CAP_DAC_READ_SEARCH` | access to `/proc/self/mem` to determine kernel version | -| `CAP_CHECKPOINT_RESTORE` | access to symlinks in the `/proc` filesystem | -| `CAP_SYS_PTRACE` | access to `/proc/pid/exe` and friends | -| `CAP_SYS_RESOURCE` | _(kernels **< 5.11** only)_ increase the amount of locked memory available. | +| `CAP_BPF` | Enables general BPF functionality and `BPF_PROG_TYPE_SOCK_FILTER` programs | +| `CAP_NET_RAW` | Used to create `AF_PACKET` raw sockets | +| `CAP_NET_ADMIN` | Required to load `BPF_PROG_TYPE_SCHED_CLS` TC programs | +| `CAP_PERFMON` | Direct packet access and pointer arithmetic and loading `BPF_PROG_TYPE_KPROBE` programs | +| `CAP_DAC_READ_SEARCH` | Access to `/proc/self/mem` to determine kernel version | +| `CAP_CHECKPOINT_RESTORE` | Access to symlinks in the `/proc` filesystem | +| `CAP_SYS_PTRACE` | Access to `/proc/pid/exe` and executable modules | +| `CAP_SYS_RESOURCE` | Increase the amount of locked memory available, **kernels < 5.11** only | | `CAP_SYS_ADMIN` | Library-level Go trace-context propagation via `bpf_probe_write_user()` and access to BTF data by the BPF metrics exporter | -**Note** Access to `CAP_PERFMON` is subject to `perf_events` access controls governed by the `kernel.perf_event_paranoid` kernel setting, which can adjusted via `sysctl` or by modifying the file `/proc/sys/kernel/perf_event_paranoid`. -The default setting for `kernel.perf_event_paranoid` is typically `2`, which is documented under the `perf_event_paranoid` section in the [kernel documentation](https://www.kernel.org/doc/Documentation/sysctl/kernel.txt) and more comprehensively under [the perf-security documentation](https://www.kernel.org/doc/Documentation/admin-guide/perf-security.rst). -Some Linux distributions define higher levels for `kernel.perf_event_paranoid`, for example Debian based distributions [also use](https://lwn.net/Articles/696216/) `kernel.perf_event_paranoid=3`, -which disallows access to `perf_event_open()` without `CAP_SYS_ADMIN`. If you are running on a distribution with `kernel.perf_event_paranoid` setting higher than `2`, -you can either modify your configuration to lower it to `2` or use `CAP_SYS_ADMIN` instead of `CAP_PERFMON`. +### Performance monitoring tasks + +Access to `CAP_PERFMON` is subject to `perf_events` access controls governed by the `kernel.perf_event_paranoid` kernel setting, which can adjusted via `sysctl` or by modifying the file `/proc/sys/kernel/perf_event_paranoid`. The default setting for `kernel.perf_event_paranoid` is typically `2`, which is documented under the `perf_event_paranoid` section in the [kernel documentation](https://www.kernel.org/doc/Documentation/sysctl/kernel.txt) and more comprehensively under [the perf-security documentation](https://www.kernel.org/doc/Documentation/admin-guide/perf-security.rst). + +Some Linux distributions define higher levels for `kernel.perf_event_paranoid`, for example Debian based distributions [also use](https://lwn.net/Articles/696216/) `kernel.perf_event_paranoid=3`, which disallows access to `perf_event_open()` without `CAP_SYS_ADMIN`. If you are running on a distribution with `kernel.perf_event_paranoid` setting higher than `2`, you can either modify your configuration to lower it to `2` or use `CAP_SYS_ADMIN` instead of `CAP_PERFMON`. + +## Example scenarios -## Examples +The following example scenarios showcases how to run Beyla as a non-root user: -The final set of required capabilities depends on the actual Beyla configuration and the type of tracers being used, as described in the section __Beyla operation modes__ above. Here are a few examples of how to run Beyla as a non-root user. +### Network metrics via a socket filter -### Network Metrics (using a socket filter) +Required capabilities: -#### Required capabilities: * `CAP_BPF` * `CAP_NET_RAW` -``` +Set the required capabilities and start Beyla: + +```bash sudo setcap cap_bpf,cap_net_raw+ep ./bin/beyla BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 bin/beyla ``` -### Network Metrics (using traffic control) +### Network metrics via traffic control + +Required capabilities: -#### Required capabilities: * `CAP_BPF` * `CAP_NET_ADMIN` * `CAP_PERFMON` -``` +Set the required capabilities and start Beyla: + +```bash sudo setcap cap_bpf,cap_net_admin,cap_perfmon+ep ./bin/beyla BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 BEYLA_NETWORK_SOURCE=tc bin/beyla ``` ### Application observability -#### Required capabilities: +Required capabilities: + * `CAP_BPF` * `CAP_DAC_READ_SEARCH` * `CAP_CHECKPOINT_RESTORE` @@ -107,14 +110,17 @@ BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 BEYLA_NETWORK_SOURCE=tc bin/ * `CAP_NET_RAW` * `CAP_SYS_PTRACE` -``` +Set the required capabilities and start Beyla: + +```bash sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace+ep ./bin/beyla -BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla +BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla ``` ### Application observability with trace context propagation -#### Required capabilities: +Required capabilities: + * `CAP_BPF` * `CAP_DAC_READ_SEARCH` * `CAP_CHECKPOINT_RESTORE` @@ -123,50 +129,57 @@ BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla * `CAP_SYS_PTRACE` * `CAP_NET_ADMIN` -``` +Set the required capabilities and start Beyla: + +```bash sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace,cap_net_admin+ep ./bin/beyla -BEYLA_BPF_ENABLE_CONTEXT_PROPAGATION=1 BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla +BEYLA_BPF_ENABLE_CONTEXT_PROPAGATION=1 BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla ``` ## Internal eBPF tracer capability requirement reference -Below is a list of internal eBPF tracers used by Beyla and their required capabilities -### Socket flow fetcher -- `CAP_BPF` -> for `BPF_PROG_TYPE_SOCK_FILTER` -- `CAP_NET_RAW` -> for creating `AF_PACKET` socket - -### Flow fetcher (tc) -- `CAP_BPF` -- `CAP_NET_ADMIN` -> for `PROG_TYPE_SCHED_CLS` -- `CAP_PERFMON` -> direct access to `struct __sk_buff::data` and pointer arithmetic - -### Watcher -- `CAP_BPF` -- `CAP_CHECKPOINT_RESTORE` -- `CAP_DAC_READ_SEARCH` -> access to `/proc/self/mem` to determine kernel version -- `CAP_PERFMON` -> for `BPF_PROG_TYPE_KPROBE` - -### Generic tracer -- `CAP_BPF` -- `CAP_DAC_READ_SEARCH` -- `CAP_CHECKPOINT_RESTORE` -- `CAP_PERFMON` -- `CAP_NET_RAW` -> for creating `AF_PACKET` socket used by `beyla_socket__http_filter` -- `CAP_SYS_PTRACE` -> access to `/proc/pid/exe` and other nodes in `/proc` - -### TC tracers +Beyla uses the following list of internal eBPF tracers with their required capabilities: + +**Socket flow fetcher:** + +* `CAP_BPF`: for `BPF_PROG_TYPE_SOCK_FILTER` +* `CAP_NET_RAW`: to create `AF_PACKET` socket + +**Flow fetcher (tc):** + +* `CAP_BPF` +* `CAP_NET_ADMIN`: for `PROG_TYPE_SCHED_CLS` +* `CAP_PERFMON`: for direct access to `struct __sk_buff::data` and pointer arithmetic + +**Watcher:** + +* `CAP_BPF` +* `CAP_CHECKPOINT_RESTORE` +* `CAP_DAC_READ_SEARCH`: for access to `/proc/self/mem` to determine kernel version +* `CAP_PERFMON`: for `BPF_PROG_TYPE_KPROBE` + +**Generic tracer:** + * `CAP_BPF` * `CAP_DAC_READ_SEARCH` +* `CAP_CHECKPOINT_RESTORE` * `CAP_PERFMON` -* `CAP_NET_ADMIN` -> for `BPF_PROG_TYPE_SCHED_CLS`, `BPF_PROG_TYPE_SOCK_OPS` and `BPF_PROG_TYPE_SK_MSG` +* `CAP_NET_RAW`: to create `AF_PACKET` socket used by `beyla_socket__http_filter` +* `CAP_SYS_PTRACE`: for access to `/proc/pid/exe` and other nodes in `/proc` + +**TC tracers:** -### GO tracer -- `CAP_BPF` -- `CAP_DAC_READ_SEARCH` -- `CAP_CHECKPOINT_RESTORE` -- `CAP_PERFMON` -- `CAP_NET_RAW` -> for creating `AF_PACKET` socket used by `beyla_socket__http_filter` -- `CAP_SYS_PTRACE` -> access to `/proc/pid/exe` and other nodes in `/proc` -- `CAP_SYS_ADMIN` -> for probe based (`bpf_probe_write_user()`) library level context propagation +* `CAP_BPF` +* `CAP_DAC_READ_SEARCH` +* `CAP_PERFMON` +* `CAP_NET_ADMIN`: for `BPF_PROG_TYPE_SCHED_CLS`, `BPF_PROG_TYPE_SOCK_OPS` and `BPF_PROG_TYPE_SK_MSG` +**GO tracer:** +* `CAP_BPF` +* `CAP_DAC_READ_SEARCH` +* `CAP_CHECKPOINT_RESTORE` +* `CAP_PERFMON` +* `CAP_NET_RAW`: to create `AF_PACKET` socket used by `beyla_socket__http_filter` +* `CAP_SYS_PTRACE`: for access to `/proc/pid/exe` and other nodes in `/proc` +* `CAP_SYS_ADMIN`: for probe based (`bpf_probe_write_user()`) library level context propagation From dea4605b7d35a2b222c9e8cde76793aca3f879aa Mon Sep 17 00:00:00 2001 From: Sean Packham Date: Mon, 3 Feb 2025 14:34:13 +0000 Subject: [PATCH 3/7] Update heading --- docs/sources/security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/security.md b/docs/sources/security.md index c0be69578..3ab89b189 100644 --- a/docs/sources/security.md +++ b/docs/sources/security.md @@ -167,7 +167,7 @@ Beyla uses the following list of internal eBPF tracers with their required capab * `CAP_NET_RAW`: to create `AF_PACKET` socket used by `beyla_socket__http_filter` * `CAP_SYS_PTRACE`: for access to `/proc/pid/exe` and other nodes in `/proc` -**TC tracers:** +**network monitoring in TC mode and context propagation:** * `CAP_BPF` * `CAP_DAC_READ_SEARCH` From 25aa41cd440f1b98d91997fea52fed8b9a6fb580 Mon Sep 17 00:00:00 2001 From: Sean Packham Date: Mon, 3 Feb 2025 14:35:05 +0000 Subject: [PATCH 4/7] Update heading --- docs/sources/security.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/security.md b/docs/sources/security.md index 3ab89b189..a91cc8453 100644 --- a/docs/sources/security.md +++ b/docs/sources/security.md @@ -158,7 +158,7 @@ Beyla uses the following list of internal eBPF tracers with their required capab * `CAP_DAC_READ_SEARCH`: for access to `/proc/self/mem` to determine kernel version * `CAP_PERFMON`: for `BPF_PROG_TYPE_KPROBE` -**Generic tracer:** +**Support for languages other than Go:** * `CAP_BPF` * `CAP_DAC_READ_SEARCH` From 67b2f9bf63904584b66abd2ea9c5ec0d5069f8b5 Mon Sep 17 00:00:00 2001 From: Sean Packham Date: Mon, 3 Feb 2025 14:41:40 +0000 Subject: [PATCH 5/7] Remove quotes and update code highlight --- docs/sources/security.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/sources/security.md b/docs/sources/security.md index a91cc8453..ba6132c59 100644 --- a/docs/sources/security.md +++ b/docs/sources/security.md @@ -14,7 +14,7 @@ aliases: # Beyla security, permissions, and capabilities -Beyla needs access to various Linux interfaces to instrument applications, such as reading from the `/proc` filesystem, loading eBPF programs, and managing network interface filters. Many of these operations require elevated permissions. The simplest solution is to run Beyla as `root`, however this might not work well in setups where full `root` access isn’t ideal. To address this, Beyla is designed to use only the specific Linux kernel capabilities needed for its current configuration. +Beyla needs access to various Linux interfaces to instrument applications, such as reading from the `/proc` filesystem, loading eBPF programs, and managing network interface filters. Many of these operations require elevated permissions. The simplest solution is to run Beyla as root, however this might not work well in setups where full root access isn’t ideal. To address this, Beyla is designed to use only the specific Linux kernel capabilities needed for its current configuration. ## Linux kernel capabilities @@ -22,7 +22,7 @@ Linux kernel capabilities are a fine-grained system for controlling access to pr Capabilities are assigned to processes and executable files. By using tools like `setcap`, administrators can assign specific capabilities to a binary, enabling it to perform only the operations it needs without running as root. For example: -```bash +```shell sudo setcap cap_net_admin,cap_net_raw+ep myprogram ``` @@ -36,7 +36,7 @@ Beyla can operate in two distinct modes: *application observability* and *networ Beyla reads its configuration and checks for the required capabilities, if any are missing it displays a warning, for example: -``` +```shell time=2025-01-27T17:21:20.197-06:00 level=WARN msg="Required system capabilities not present, Beyla may malfunction" error="the following capabilities are required: CAP_DAC_READ_SEARCH, CAP_BPF, CAP_CHECKPOINT_RESTORE" ``` @@ -79,7 +79,7 @@ Required capabilities: Set the required capabilities and start Beyla: -```bash +```shell sudo setcap cap_bpf,cap_net_raw+ep ./bin/beyla BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 bin/beyla ``` @@ -94,7 +94,7 @@ Required capabilities: Set the required capabilities and start Beyla: -```bash +```shell sudo setcap cap_bpf,cap_net_admin,cap_perfmon+ep ./bin/beyla BEYLA_NETWORK_METRICS=1 BEYLA_NETWORK_PRINT_FLOWS=1 BEYLA_NETWORK_SOURCE=tc bin/beyla ``` @@ -112,7 +112,7 @@ Required capabilities: Set the required capabilities and start Beyla: -```bash +```shell sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace+ep ./bin/beyla BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla ``` @@ -131,7 +131,7 @@ Required capabilities: Set the required capabilities and start Beyla: -```bash +```shell sudo setcap cap_bpf,cap_dac_read_search,cap_perfmon,cap_net_raw,cap_sys_ptrace,cap_net_admin+ep ./bin/beyla BEYLA_BPF_ENABLE_CONTEXT_PROPAGATION=1 BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER=text bin/beyla ``` From a65c725c9a90c34888f143c4b9b263fa4046fcd0 Mon Sep 17 00:00:00 2001 From: Rafael Roquetto Date: Wed, 5 Feb 2025 17:56:01 -0600 Subject: [PATCH 6/7] Review feedback - add further context --- docs/sources/security.md | 53 +++++++++++++++++--------------- docs/sources/setup/kubernetes.md | 6 +--- 2 files changed, 29 insertions(+), 30 deletions(-) diff --git a/docs/sources/security.md b/docs/sources/security.md index ba6132c59..524b18412 100644 --- a/docs/sources/security.md +++ b/docs/sources/security.md @@ -30,6 +30,8 @@ This example grants the `CAP_NET_ADMIN` and `CAP_NET_RAW` capabilities to `mypro By carefully choosing and assigning capabilities you can lower the risk of privilege escalation while still letting processes do what they need to. +More information can be found in the [capabilities manual page](https://man7.org/linux/man-pages/man7/capabilities.7.html). + ## Beyla operation modes Beyla can operate in two distinct modes: *application observability* and *network observability*. These modes are not mutually exclusive and can be used together as needed. For more details on enabling these modes, refer to the [configuration documentation](/docs/beyla/latest/configure/options/). @@ -48,17 +50,17 @@ You can set `BEYLA_ENFORCE_SYS_CAPS=1`, which causes Beyla to fail immediately i Beyla requires the following The following table Below is a list of capabilities and their usage in the context of Beyla -| Capability | Usage in Beyla | -| ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | -| `CAP_BPF` | Enables general BPF functionality and `BPF_PROG_TYPE_SOCK_FILTER` programs | -| `CAP_NET_RAW` | Used to create `AF_PACKET` raw sockets | -| `CAP_NET_ADMIN` | Required to load `BPF_PROG_TYPE_SCHED_CLS` TC programs | -| `CAP_PERFMON` | Direct packet access and pointer arithmetic and loading `BPF_PROG_TYPE_KPROBE` programs | -| `CAP_DAC_READ_SEARCH` | Access to `/proc/self/mem` to determine kernel version | -| `CAP_CHECKPOINT_RESTORE` | Access to symlinks in the `/proc` filesystem | -| `CAP_SYS_PTRACE` | Access to `/proc/pid/exe` and executable modules | -| `CAP_SYS_RESOURCE` | Increase the amount of locked memory available, **kernels < 5.11** only | -| `CAP_SYS_ADMIN` | Library-level Go trace-context propagation via `bpf_probe_write_user()` and access to BTF data by the BPF metrics exporter | +| Capability | Usage in Beyla | +| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `CAP_BPF` | Enables general BPF functionality and socket filter (`BPF_PROG_TYPE_SOCK_FILTER`) programs, used for capturing network flows in *network observability mode*. | +| `CAP_NET_RAW` | Used to create `AF_PACKET` raw sockets, which is the mechanism used to attach socket filter programs used for capturing network flows in *network observability mode*. | +| `CAP_NET_ADMIN` | Required to load `BPF_PROG_TYPE_SCHED_CLS` TC programs - these programs are used for capturing network flows and for trace context propagation, both for *network and application observability*. | +| `CAP_PERFMON` | Used for trace context propagation, general *application observability* and network flow monitoring. Allows direct packet access by TC programs, loading eBPF probes in the kernel and pointer arithmetic used by these programs. | +| `CAP_DAC_READ_SEARCH` | Access to `/proc/self/mem` to determine kernel version, used by Beyla to determine the appropriate set of supported features to enable. | +| `CAP_CHECKPOINT_RESTORE` | Access to symlinks in the `/proc` filesystem, used by Beyla to obtain various process and system information. | +| `CAP_SYS_PTRACE` | Access to `/proc/pid/exe` and executable modules, used by Beyla to scan executable symbols and instrument different parts of a program. | +| `CAP_SYS_RESOURCE` | Increase the amount of locked memory available, **kernels < 5.11** only | +| `CAP_SYS_ADMIN` | Library-level Go trace-context propagation via `bpf_probe_write_user()` and access to BTF data by the BPF metrics exporter | ### Performance monitoring tasks @@ -138,48 +140,49 @@ BEYLA_BPF_ENABLE_CONTEXT_PROPAGATION=1 BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER= ## Internal eBPF tracer capability requirement reference -Beyla uses the following list of internal eBPF tracers with their required capabilities: +The internal architecture of Beyla is made of different "tracers" - a set of eBPF programs that implement the underlying functionality used by Beyla. A tracer may load and use different kinds of eBPF programs, each requiring their own set of capabilities. +The list below maps each internal tracer to their required capabilities, intended to serve as a reference for developers, contributors, and those interested in the internals of Beyla: -**Socket flow fetcher:** +**(Network observability) Socket flow fetcher:** * `CAP_BPF`: for `BPF_PROG_TYPE_SOCK_FILTER` -* `CAP_NET_RAW`: to create `AF_PACKET` socket +* `CAP_NET_RAW`: to create `AF_PACKET` socket and attaching socket filters to a network interface -**Flow fetcher (tc):** +**(Network observability) Flow fetcher (tc):** * `CAP_BPF` -* `CAP_NET_ADMIN`: for `PROG_TYPE_SCHED_CLS` -* `CAP_PERFMON`: for direct access to `struct __sk_buff::data` and pointer arithmetic +* `CAP_NET_ADMIN`: for loading `PROG_TYPE_SCHED_CLS` eBPF TC programs, used for inspecting network traffic +* `CAP_PERFMON`: for direct access to packet memory via `struct __sk_buff::data` and to allow pointer arithmetic in eBPF programs -**Watcher:** +**(Application observability) Watcher:** * `CAP_BPF` * `CAP_CHECKPOINT_RESTORE` * `CAP_DAC_READ_SEARCH`: for access to `/proc/self/mem` to determine kernel version -* `CAP_PERFMON`: for `BPF_PROG_TYPE_KPROBE` +* `CAP_PERFMON`: for loading `BPF_PROG_TYPE_KPROBE` eBPF programs that require pointer arithmetic -**Support for languages other than Go:** +**(Application observability) Support for languages other than Go:** * `CAP_BPF` * `CAP_DAC_READ_SEARCH` * `CAP_CHECKPOINT_RESTORE` * `CAP_PERFMON` -* `CAP_NET_RAW`: to create `AF_PACKET` socket used by `beyla_socket__http_filter` +* `CAP_NET_RAW`: to create `AF_PACKET` socket used to attach `beyla_socket__http_filter` to network interfaces * `CAP_SYS_PTRACE`: for access to `/proc/pid/exe` and other nodes in `/proc` -**network monitoring in TC mode and context propagation:** +**(Application and network observability) network monitoring in TC mode and context propagation:** * `CAP_BPF` * `CAP_DAC_READ_SEARCH` * `CAP_PERFMON` -* `CAP_NET_ADMIN`: for `BPF_PROG_TYPE_SCHED_CLS`, `BPF_PROG_TYPE_SOCK_OPS` and `BPF_PROG_TYPE_SK_MSG` +* `CAP_NET_ADMIN`: allows loading`BPF_PROG_TYPE_SCHED_CLS`, `BPF_PROG_TYPE_SOCK_OPS` and `BPF_PROG_TYPE_SK_MSG`, all used by trace context propagation and network monitoring -**GO tracer:** +**(Application observabilty) GO tracer:** * `CAP_BPF` * `CAP_DAC_READ_SEARCH` * `CAP_CHECKPOINT_RESTORE` * `CAP_PERFMON` -* `CAP_NET_RAW`: to create `AF_PACKET` socket used by `beyla_socket__http_filter` +* `CAP_NET_RAW`: to create `AF_PACKET` socket used to attach `beyla_socket__http_filter` to network interfaces * `CAP_SYS_PTRACE`: for access to `/proc/pid/exe` and other nodes in `/proc` * `CAP_SYS_ADMIN`: for probe based (`bpf_probe_write_user()`) library level context propagation diff --git a/docs/sources/setup/kubernetes.md b/docs/sources/setup/kubernetes.md index a248c1121..9013a3ccc 100644 --- a/docs/sources/setup/kubernetes.md +++ b/docs/sources/setup/kubernetes.md @@ -251,11 +251,7 @@ In all of the examples so far, `privileged:true` or the `SYS_ADMIN` Linux capabi The following guide is based on tests performed mainly by running `containerd` with `GKE`, `kubeadm`, `k3s`, `microk8s` and `kind`. -To run Beyla unprivileged, you need to run a `privileged` init container which performs setup tasks which require elevated privileges. Then you need to replace the `privileged:true` setting with a set of Linux [capabilities](https://www.man7.org/linux/man-pages/man7/capabilities.7.html). A comprehensive list of capabilities required by Beyla can be found in [Security, permissions and capabilities]({{< relref "../security" >}}). - -In addition to these Linux capabilities, many Kubernetes versions include [AppArmour](https://kubernetes.io/docs/tutorials/security/apparmor/), which tough policies adds additional restrictions to unprivileged containers. By [default](https://github.com/moby/moby/blob/master/profiles/apparmor/template.go), the AppArmour policy restricts the use of `mount` and the access to `/sys/fs/` directories. Beyla requires access to `/sys/fs/cgroup/`. - -Because of the AppArmour restriction, to run Beyla as unprivileged container, you may need to set `container.apparmor.security.beta.kubernetes.io/beyla: "unconfined"` in your Kubernetes deployment files in case you run into issues. +To run Beyla unprivileged, you need to replace the `privileged:true` setting with a set of Linux [capabilities](https://www.man7.org/linux/man-pages/man7/capabilities.7.html). A comprehensive list of capabilities required by Beyla can be found in [Security, permissions and capabilities]({{< relref "../security" >}}). **Note** Loading BPF programs requires that Beyla is able to read the Linux performance events, or at least be able to execute the Linux Kernel API `perf_event_open()`. From 9407e752b9804af969aa9eefcb3ab3ee70afbf82 Mon Sep 17 00:00:00 2001 From: Sean Packham Date: Thu, 6 Feb 2025 13:36:51 +0000 Subject: [PATCH 7/7] Edits --- docs/sources/security.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/sources/security.md b/docs/sources/security.md index 524b18412..23f7b5e8a 100644 --- a/docs/sources/security.md +++ b/docs/sources/security.md @@ -48,7 +48,7 @@ You can set `BEYLA_ENFORCE_SYS_CAPS=1`, which causes Beyla to fail immediately i ## List of capabilities required by Beyla -Beyla requires the following The following table Below is a list of capabilities and their usage in the context of Beyla +Beyla requires the following list of capabilities for its functionality: | Capability | Usage in Beyla | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -140,7 +140,9 @@ BEYLA_BPF_ENABLE_CONTEXT_PROPAGATION=1 BEYLA_OPEN_PORT=8080 BEYLA_TRACE_PRINTER= ## Internal eBPF tracer capability requirement reference -The internal architecture of Beyla is made of different "tracers" - a set of eBPF programs that implement the underlying functionality used by Beyla. A tracer may load and use different kinds of eBPF programs, each requiring their own set of capabilities. +Beyla uses *tracers*, a set of eBPF programs that implement the underlying functionality. +A tracer may load and use different kinds of eBPF programs, each requiring their own set of capabilities. + The list below maps each internal tracer to their required capabilities, intended to serve as a reference for developers, contributors, and those interested in the internals of Beyla: **(Network observability) Socket flow fetcher:** @@ -177,7 +179,7 @@ The list below maps each internal tracer to their required capabilities, intende * `CAP_PERFMON` * `CAP_NET_ADMIN`: allows loading`BPF_PROG_TYPE_SCHED_CLS`, `BPF_PROG_TYPE_SOCK_OPS` and `BPF_PROG_TYPE_SK_MSG`, all used by trace context propagation and network monitoring -**(Application observabilty) GO tracer:** +**(Application observability) GO tracer:** * `CAP_BPF` * `CAP_DAC_READ_SEARCH`