Introduction to BPF Manager (bpfman)

Original Article: https://www.ebpf.top/post/bpfman_fedora_40

1. Background

Fedora 40 proposes bpfman as the default program manager. The open-source project bpfman provides a deeper understanding of the eBPF runtime state, making it easier to manage eBPF programs (including loading, unloading, and viewing runtime status). This proposal requires approval from the Fedora Engineering Steering Committee (FESCo), but if successful, bpfman is likely to appear in Fedora 40 in April to enhance eBPF management.

So, what exactly is bpfman? This article will give you a brief introduction to bpfman and its working principles.

2. Introducing bpfman

Originally named bpfd, bpfman is developed based on the Rust Aya library, using the Rust programming language. The bpfman project is designed to simplify the loading, unloading, modification, and monitoring of eBPF programs on Kubernetes clusters or single hosts. The bpfman project consists of the bpfman daemon, eBPF CRD (Custom Resource Definition), bpfman-agent, and bpfman-operator. Among them, CRDs, bpfman-agent, and bpfman-operator are components deployed in a distributed manner around the Kubernetes environment:

• bpfman: A system daemon running on a single host, it provides a gRPC API for loading, unloading, modifying, and monitoring eBPF programs.
• eBPF CRDs: Provides two types of CRDs: deployment CRDs and BPF program status view CRDs. Deployment-related CRDs (such as XdpProgram and TcProgram) are used to load different types of eBPF programs. Program status view CRDs (BpfProgram) are used to manage the runtime status of eBPF programs loaded into the kernel.
• bpfman-agent: Runs in a container within the bpfman daemon and ensures that the selected node’s eBPF programs are in the desired state.
• bpfman-operator: An Operator built using the Operator SDK, it manages the installation and lifecycle of bpfman-agent and CRDs in the Kubernetes cluster.

3. Standalone Deployment Process

When deploying bpfman in a standalone environment, the workflow is as follows. The bpfman binary can be deployed as both a server and a client program. In addition to using the bpfman client program, users can also interact with the bpfman server for various management operations in user space programs:

1. When the go-xdp-counter user space program starts, it sends a gRPC request to bpfman through a Unix socket to request the loading of the go-xdp-counter eBPF bytecode (bpfman/examples/go-xdp-counter/bpf_bpfel.o) located on the disk, with a priority of 50, and the program applies to the ens3 interface.
2. Upon receiving the request, bpfman loads the corresponding eBPF program (go-xdp-counter) and returns the UUID of the running program.
3. Users can use the bpfman list command to display the eBPF programs managed by the system.
4. After the eBPF bytecode (go-xdp-counter) is successfully loaded, the running eBPF program will count the packets and bytes and write them into a shared map structure.
5. The corresponding user space program (go-xdp-counter) periodically reads the counters from the shared map and records the values.

The loaded bytecode can also be a remote image, such as sudo ./go-xdp-counter -iface ens3 -image quay.io/bpfman-bytecode/go-xdp-counter:latest. The bytecode image needs to follow certain specifications.

4. Kubernetes Cluster Deployment Process

Deploying and managing eBPF programs in a distributed environment like a Kubernetes cluster can be more complex. bpfman focuses on addressing the following challenges:

• Managing the permissions and capabilities required for enabling eBPF applications.
• Resolving the issue of multiple eBPF programs on the same hook function and their lifecycle management (especially multiple mounts under xdp mode, which is not supported by the kernel by default).
• Simplifying the complexity of deployment and program lifecycle in Kubernetes.
• Aimed at optimization in terms of security, visibility, multi-program support, and productivity.

Deploying and managing eBPF programs with bpfman in a Kubernetes environment can be divided into two stages:

1. Managing eBPF bytecode programs running in the kernel.
2. Deploying and reading data from user space programs corresponding to the kernel space (usually used to read runtime data or load configurations), which need to be deployed separately (mounted and read data through CSI).

The deployment and management process of bpfman in Kubernetes is shown in the following diagram:

1. Write a CR resource type XdpProgram, define parameters related to deployment machine labels and eBPF bytecode (such as interface, priority, and BPF bytecode image). In this example, the resource name of XdpProgram is go-xdp-counter-example. Then, you can use kbueclt apply -f bytecode.yaml to submit it to the cluster to take effect. The complete yaml file is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# cat examples/config/base/go-xdp-counter/bytecode.yaml
apiVersion: bpfman.io/v1alpha1
kind: XdpProgram
metadata:
  labels:
    app.kubernetes.io/name: xdpprogram
  name: go-xdp-counter-example
spec:
  name: xdp_stats
  # Select all nodes
  nodeselector: {}
  interfaceselector:
    primarynodeinterface: true
  priority: 55
  bytecode:
    image:
      url: quay.io/bpfman-bytecode/go-xdp-counter:latest

2. bpfman-agent runs on the expected nodes and monitors the corresponding CR resource object XdpProgram. The bpfman-agent ensures that a corresponding BpfProgram object is generated for the created or modified XdpProgram object. The name of the BpfProgram object is a combination of the XdpProgram name, node name, interface, and connection point information.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

   $ kubectl get xdpprogram NAME PRIORITY DIRECTION go-xdp-counter-example 55 $ kubectl get xdpprograms go-xdp-counter-example -o yaml apiVersion: bpfman.io/v1alpha1 kind: XdpProgram ... status: conditions: - lastTransitionTime: "2023-11-06T21:05:21Z" message: bpfProgramReconciliation Succeeded on all nodes reason: ReconcileSuccess status: "True" type: ReconcileSuccess

   In the above XdpProgram object’s status field, we can see that the program has been successfully distributed to the target nodes.

3. bpfman-agent loads or unloads eBPF bytecode programs as needed by making gRPC calls to bpfman. The behavior of bpfman is the same as described in the single-machine running example.

4. Finally, bpfman-agent updates the status of the BpfProgram object.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

   $ kubectl get bpfprograms NAME AGE go-xdp-counter-example-bpfman-deployment-control-plane-eth0 60m $ kubectl get go-xdp-counter-example-bpfman-deployment-control-plane-eth0 -o yaml apiVersion: bpfman.io/v1alpha1 kind: BpfProgram .... spec: type: xdp status: conditions: - lastTransitionTime: "2023-11-06T21:05:21Z" message: Successfully loaded bpfProgram reason: bpfmanLoaded status: "True" type: Loaded

5. bpfman-operator monitors all BpfProgram objects and updates the status of XdpProgram objects to show whether the eBPF program has been applied to the corresponding nodes.

However, attentive readers may have noticed that the above process only completes the loading of eBPF bytecode into the kernel (with bpfman acting as the eBPF loader), but the runtime-exposed data and metrics have not yet been collected. Therefore, a separate userspace program needs to be deployed to read the data generated by the eBPF counter program (as shown in the above diagram using a map data structure).

Here, we need to deploy a separate userspace program to consume the map data generated by the eBPF counter program. bpfman provides a Container Storage Interface (CSI) driver to expose eBPF maps to userspace containers. To avoid having to mount the host directory containing the fixed files of the map into the container and forcing the container to have access to that host directory, the CSI driver mounts the map to a specified location inside the container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#cat config/base/go-xdp-counter/deployment.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: go-xdp-counter-ds
  namespace: go-xdp-counter
  labels:
    k8s-app: go-xdp-counter
spec:
  :
  template:
    :
    spec:
       :
      containers:
      - name: go-xdp-counter
        :
        volumeMounts:
        - name: go-xdp-counter-maps                        <==== 2) VolumeMount in container
          mountPath: /run/xdp/maps                         <==== 2a) Mount path in the container
          readOnly: true
      volumes:
      - name: go-xdp-counter-maps                          <==== 1) Volume describing the map
        csi:
          driver: csi.bpfman.io                             <==== 1a) bpfman CSI Driver
          volumeAttributes:
            csi.bpfman.io/program: go-xdp-counter-example   <==== 1b) eBPF Program owning the map
            csi.bpfman.io/maps: xdp_stats_map               <==== 1c) Map to be exposed to the container

Here, we introduced the complete process of deploying xdp-type programs using bpfman in a Kubernetes environment. Compared to deploying on a single machine, management in a distributed environment is more complex.

5. Summary

Here we provided a brief introduction to bpfman. For more details, please refer to https://bpfman.io/. Based on my personal use and testing experience, here are a few points I would like to share:

• bpfman is developed using Rust and is somewhat niche in the cloud-native field.

• In the K8s environment, eBPF loading and data reading are separate processes, mounted through the CSI interface. Both deployment installation and usage are not very smooth, and there may be better solutions in the future.

• The project uses the libxdp library to implement the loading of multiple XDP programs on a single interface and deployment without a daemon, which can be considered as a special feature.

In any case, bpfman provides a complete architectural implementation for managing and distributing eBPF programs in a single machine or Kubernetes distributed environment, which is worth learning.

Appendix: bpfman Single Machine Verification

It can be directly installed via the rpm package in the Fedora distribution. In the Ubuntu system, it needs to be installed from source code.

Development Environment Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
  stable-x86_64-unknown-linux-gnu installed - rustc 1.75.0 (82e1608df 2023-12-21)

.....
Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source "$HOME/.cargo/env"

$ source "$HOME/.cargo/env"

$ rustup toolchain install nightly -c rustfmt,clippy,rust-src
...
info: installing component 'rustfmt'

  nightly-x86_64-unknown-linux-gnu installed - rustc 1.77.0-nightly (5518eaa94 2024-01-29)

info: checking for self-update

Code Download and Compilation of bpfman

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

$ git clone https://github.com/bpfman/bpfman.git && cd bpfman
$ cargo xtask build-ebpf --libbpf-dir=/home/dave/test/libbpf-bootstrap-1/libbpf

# Build bpfman, the compiled executable can be found at ./target/debug/bpfman
$ cargo build

$ file ./target/debug/bpfman
./target/debug/bpfman: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=e3028a97943b1dafcadbd7c9e9c99d9fb49aa5fe, for GNU/Linux 3.2.0, with debug_info, not stripped


$ ./target/debug/bpfman --help
An eBPF manager focusing on simplifying the deployment and administration of eBPF programs.

Usage: bpfman <COMMAND>
Commands:
  load    Load an eBPF program from a local .o file
  unload  Unload an eBPF program using the program id
  list    List all eBPF programs loaded via bpfman
  get     Get an eBPF program using the program id
  image   eBPF Bytecode Image related commands
  system  Run bpfman as a service
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help
          Print help (see a summary with '-h')

# Run as a standalone process
$ RUST_LOG=info ./target/debug/bpfman system service --timeout 0

# Or install as a system service
$ sudo ./scripts/setup.sh install
Copy CLI TAB Completion files:
  CLI TAB Completion files not generated yet. Use "cargo xtask build-completion" to generate.
Copy Manpage files:
  CLI Manpage files not generated yet. Use "cargo xtask build-man-page" to generate.
Copy binaries:
  Copying "../target/debug/bpfman" to "/usr/sbin"
Copy service files:
  Copying "bpfman.socket"
  Copying "bpfman.service"
  Starting "bpfman.socket"
Created symlink /etc/systemd/system/sockets.target.wants/bpfman.socket → /lib/systemd/system/bpfman.socket.

$ sudo  systemctl start bpfman
$ sudo systemctl status bpfman
● bpfman.service - Run bpfman as a service
     Loaded: loaded (/lib/systemd/system/bpfman.service; static)
     Active: active (running) since Tue 2024-01-30 16:10:28 CST; 2s ago
TriggeredBy: ● bpfman.socket
   Main PID: 1563623 (bpfman)
      Tasks: 23 (limit: 18814)
     Memory: 25.4M
        CPU: 237ms
     CGroup: /system.slice/bpfman.service
             └─1563623 /usr/sbin/bpfman system service

Jan 30 16:10:28 dave systemd[1]: Started Run bpfman as a service.
Jan 30 16:10:28 dave bpfman[1563623]: Log using journald
Jan 30 16:10:28 dave bpfman[1563623]: Has CAP_BPF: true
Jan 30 16:10:28 dave bpfman[1563623]: Has CAP_SYS_ADMIN: true
....

Compile and load eBPF programs

1
2
3
4
5
6

$ sudo apt-get install libbpf-dev
$ sudo go install github.com/cilium/ebpf/cmd/bpf2go@master

$ cd bpfman/examples/
$ make generate
$ make build

Test program management functionality

1
2
3
4
5
6
7

# cd bpfman/examples/go-xdp-counter
# ./go-xdp-counter -iface lo
2024/01/31 19:28:25 Using Input: Interface=lo Priority=50 Source=/home/dave/bpfman/examples/go-xdp-counter/bpf_bpfel.o
2024/01/31 19:28:30 Program registered with id 1872
2024/01/31 19:28:33 153212 packets received
2024/01/31 19:28:33 9993784 bytes received
...

In addition to running user space programs, you can also load them using bpfman load (supports loading from images).

You can check the loaded logs in the bpfman process log:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# RUST_LOG=info ./target/debug/bpfman system service --timeout 0
[INFO  bpfman::cli::system] Log using env_logger
[INFO  bpfman::cli::system] Has CAP_BPF: true
[INFO  bpfman::cli::system] Has CAP_SYS_ADMIN: true
[INFO  bpfman::serve] Using no inactivity timer
[INFO  bpfman::serve] Using default Unix socket
[INFO  bpfman::serve] Listening on /run/bpfman/sock/bpfman.sock
[INFO  bpfman::oci_utils::cosign] Starting Cosign Verifier, downloading data from Sigstore TUF repository```
[INFO  bpfman::command] Loading program bytecode from file: /home/dave/bpfman/examples/go-xdp-counter/bpf_bpfel.o
[INFO  bpfman::oci_utils::cosign] The bytecode image: quay.io/bpfman/xdp-dispatcher:v2 is signed
[INFO  bpfman::bpf] Added xdp program with name: xdp_stats and id: 1872

To view the loaded programs, you can use bpfman list:

1
2
3

# bpfman list
 Program ID  Name       Type  Load Time
 1872        xdp_stats  xdp   2024-01-31T19:28:30+0800

To uninstall, simply run bpfman unload 1872.

Or you can use bpfman get pid to view specific execution details:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# bpfman get 1872
 Bpfman State
 Name:          xdp_stats
 Path:          /home/dave/bpfman/examples/go-xdp-counter/bpf_bpfel.o
 Global:        None
 Metadata:      None
 Map Pin Path:  /run/bpfman/fs/maps/1872
 Map Owner ID:  None
 Maps Used By:  1872
 Priority:      50
 Iface:         lo
 Position:      0
 Proceed On:    pass, dispatcher_return

 Kernel State
 ID:                               1872
 Name:                             xdp_stats
 Type:                             xdp
 Loaded At:                        2024-01-31T19:28:30+0800
 Tag:                              4d23a1d7f3618653
 GPL Compatible:                   true
 Map IDs:                          [468]
 BTF ID:                           598
 Size Translated (bytes):          168
 JITted:                           true
 Size JITted:                      104
 Kernel Allocated Memory (bytes):  4096
 Verified Instruction Count:       21

Note: The translation provided is a colloquial, professional, elegant, and fluent interpretation of the text.