eBPF in Kubernetes: Improving Observability for K8s Clusters

What Is eBPF? 

Extended Berkeley Packet Filter (eBPF) is an innovative technology derived from the Linux kernel that allows sandboxed programs to run within the operating system kernel. It can safely and efficiently extend the kernel’s functionality without loading kernel modules or changing the source code.

Operating systems have traditionally been the best place to implement security, observability, and networking features, given the kernel’s authority to monitor and control the entire system. The operating system kernel is resistant to evolution due to its key role and high requirements for security and stability. As a result, operating system-level innovation has been slow compared to features implemented externally.

eBPF radically changes this formula by allowing sandboxed programs to run in the operating system. Application development teams can use eBPF programs to provide additional functionality at runtime. The operating system uses a just-in-time (JIT) compiler and validation engine to ensure secure and efficient execution, similar to native compilation. 

eBPF and Kubernetes

Monitoring and scaling Kubernetes clusters have traditionally been challenging, due to the distributed architecture of Kubernetes, its complexity, and the abstract nature of Kubernetes metrics. Kubernetes administrators lack visibility into resource consumption at the kernel level, making scaling and fine-tuning harder. eBPF can improve observability by collecting granular data at the kernel level across a Kubernetes cluster.

In this article:

• How eBPF Works
• eBPF Use Cases Examples
• Networking with eBPF
• Security with eBPF
• eBPF for Tracing and Profiling
• Using eBPF Programs for Kubernetes

How eBPF Works 

An eBPF program is an event-driven solution tied to a specific code path. This code path includes triggers called hooks that, when passed, execute the associated eBPF program. Examples of hooks include system calls, network events, kernel tracepoints, and function entries. 

When triggered, program code is first compiled to Berkeley Packet Filter (BPF) bytecode. In turn, the bytecode is verified before it runs, to ensure it doesn’t create a loop. This step prevents the program from compromising the Linux kernel either accidentally or on purpose.

The program makes a helper call after the hook triggers it. A helper call is a function that provides eBPF with a memory access feature. The kernel must predefine each helper call, although the list of possible functions can continue to grow.

Organizations originally used eBPF to improve security and observability while filtering network packets. However, it has become a safer, more convenient, and highly performant way to implement user-supplied code.

eBPF Use Cases Examples 

eBPF has important advantages for several use cases.

Networking with eBPF

Linux offers a default network path for forwarding packets to their destination. All packets must go through this path. This can make it difficult to optimize performance and customize traffic behavior, especially in complex micoservices environments.

eBPF programs can provide enhanced networking capabilities, allowing packets to bypass the default path and travel directly to their intended destination. eBPF can also apply network policies to route packets and improve control over traffic.

Security with eBPF 

eBPF makes it possible to  view and interpret system calls in real time. It also enables visibility at the socket and packet levels. 

Unlike traditional security systems, which handle system calls, processes, and network traffic separately, security systems based on eBPF combine visibility and control in a single system to manage all networking and system events. This makes it much easier to identify suspicious events, with improved control and deeper context.

eBPF for Tracing and Profiling

Attaching an eBPF program to tracepoints, user application points, and kernel probe points provides deeper visibility into the system’s behavior and each application’s performance at runtime. 

It provides introspection at the application and system levels, allowing teams to combine these two views. This capability provides powerful, unique insights for troubleshooting system performance problems. An advanced statistical data structure enables the efficient extraction of actionable visibility information without exporting the large volumes of sampling data required by other similar systems.

Using eBPF Programs for Kubernetes 

Monitoring and scaling clusters have traditionally been challenging for the following reasons: 

• Kubernetes’ distributed architecture—clusters usually include multiple nodes running different container types, making it harder to collect logs and metrics across this distributed environment.
• Kubernetes’ complexity—Kubernetes has many components and services generating different metrics, making it harder to track overall performance. 
• Metric abstraction—Kubernetes reports high-level event data and metrics but doesn’t record the specific processes within each cluster. This abstraction makes it harder to understand the inner workings of clusters.

Kubernetes administrators often struggle to provision the right number of resources for each cluster. They lack visibility into resource consumption at the kernel level, making scaling and fine-tuning harder. Over-provisioning clusters can waste money, while under-provisioning them can result in performance issues. 

Related content: Read our guide to Kubernetes architecture

An eBPF program can collect granular data at the kernel level across the Kubernetes cluster, allowing teams to scale clusters more easily. Extended Berkeley Packet Filter offers the following benefits:

• Service-level monitoring—monitoring processes in specific services is easier, allowing teams to track resource consumption patterns. 
• Granularity—eBPF programs provide more detailed information than logs, improving overall visibility. 
• Easy deployment—eBPF doesn’t require special instrumentation like the sidecar containers used by log agents. It works at the operating system level, consuming fewer resources with minimal overhead. 

Kubernetes admins use eBPF to collect detailed monitoring data more easily, but they still have to scale the clusters. eBPF won’t do this automatically; it only provides the insights to help make scaling decisions. Another potential challenge is orchestration—admins must deploy eBPF on every node in each cluster, and no tools are available to centralize monitoring across a server cluster. 

A third-party management tool can make Kubernetes orchestration easier by using eBPF internally. For instance, the open source networking  tool Cilium and proprietary platforms like Kentik and Splunk use eBPF. These tools automatically deploy monitoring agents in clusters and help teams understand the eBPF data, so teams can leverage eBPF without writing a program from scratch.