Observability has become an essential practice in modern software engineering. As distributed systems grow in complexity, so does the need for robust monitoring and observability tools. One technology that’s been making waves in this field is eBPF (extended Berkeley Packet Filter). Originally designed for network packet filtering, eBPF has evolved into a powerful tool for monitoring and improving observability in Linux systems. It enables users to run sandboxed programs within the kernel, providing deep insights into the system without modifying source code or kernel modules.
In this article, we will explore how eBPF works, how it can be used to enhance observability, and provide hands-on examples of how to utilize it in real-world applications.
What is eBPF?
eBPF is a kernel technology that allows for the safe execution of custom programs within the Linux kernel. These programs run in a sandboxed environment, meaning they cannot crash the kernel or interfere with its normal operation. With eBPF, you can instrument and observe system calls, network activity, kernel functions, and user-space processes without altering the kernel itself.
The magic of eBPF lies in its ability to run programs at various hook points within the Linux kernel. These programs can be attached to networking, file systems, process events, or other kernel subsystems. Because eBPF programs are verified before execution, they are safe to run without causing kernel panics or introducing security vulnerabilities.
Why Use eBPF for Observability?
Traditional observability tools often come with limitations, including high overhead, limited visibility, or the need for intrusive modifications to the system. eBPF addresses many of these challenges with the following advantages:
- Low Overhead: eBPF programs run directly within the kernel space, minimizing context switches and resource consumption.
- Deep Visibility: You can monitor system calls, network packets, CPU cycles, and other internal operations at a very granular level.
- Dynamic Instrumentation: eBPF allows for live instrumentation without stopping services or modifying kernel modules.
- Security: Since eBPF programs run in a restricted sandbox, they cannot perform unauthorized actions or interfere with the system’s stability.
eBPF Use Cases in Observability
Here are some common use cases where eBPF can shine in observability:
- System Call Tracing: With eBPF, you can trace system calls such as file I/O operations, network communication, or memory allocation to understand how processes interact with the kernel.
- Network Monitoring: eBPF can provide insights into packet-level data, including TCP retransmissions, dropped packets, and bandwidth consumption.
- Performance Profiling: By attaching eBPF programs to performance-critical functions, you can gather real-time performance metrics like CPU usage, cache misses, and execution time.
- Security Monitoring: eBPF can detect suspicious activities, such as unauthorized file accesses or anomalous network traffic, enhancing your system’s security.
Let’s take a closer look at how to use eBPF for observability with coding examples.
Setting Up eBPF for Observability
To get started with eBPF, you’ll need the following tools:
- bcc (BPF Compiler Collection): A set of tools and Python bindings that make it easier to write eBPF programs.
- BPFtrace: A higher-level tracing tool that simplifies the writing of eBPF programs for tracing events.
- Linux Kernel Version: eBPF is available in Linux kernels 4.x and later. Make sure your system is running an appropriate kernel version.
Here’s how you can set up these tools on your system:
# Install the necessary dependencies
sudo apt-get update
sudo apt-get install -y bpfcc-tools linux-headers-$(uname -r) \
bpftrace bccbpftrace -e ‘BEGIN { printf(“Hello, eBPF!\n”); }’
Tracing System Calls with eBPF
Let’s start by tracing system calls. For this example, we will trace the open system call to monitor file accesses.
BPFtrace Example
BPFtrace simplifies eBPF programming by providing a high-level scripting interface. Here’s how you can use it to trace the open system call:
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("File opened: %s\n", str(args->filename)); }'
This script attaches to the sys_enter_openat tracepoint, which is triggered whenever a file is opened on the system. It prints the name of the file being opened.
Output:
File opened: /etc/hostname
File opened: /var/log/syslog
File opened: /home/user/.bashrc
bcc Example
You can also use the bcc library to write more complex eBPF programs in Python. Here’s an example that traces the execve system call:
from bcc import BPF
# BPF program to trace execve system call
bpf_program = “””
TRACEPOINT_PROBE(syscalls, sys_enter_execve) {
bpf_trace_printk(“Process executed: %s\\n”, args->argv[0]);
return 0;
}
“””
# Load BPF program
b = BPF(text=bpf_program)
# Print trace output
b.trace_print()
This script monitors the execution of processes on the system and prints the name of each program that is executed.
Output:
Process executed: /bin/bash
Process executed: /usr/bin/python3
Process executed: /bin/ls
Network Monitoring with eBPF
eBPF can also be used for advanced network monitoring. Let’s track dropped network packets using BPFtrace.
sudo bpftrace -e 'tracepoint:net:net_dev_xmit { if (args->ret != 0) { printf("Packet dropped: %d\n", args->ret); } }'
This script attaches to the net_dev_xmit tracepoint, which is triggered whenever a network packet is transmitted. If the packet is dropped, the return value (args->ret) will be non-zero, and the packet drop event will be printed.
Output:
Packet dropped: -5
Packet dropped: -11
Performance Profiling with eBPF
Let’s use eBPF to monitor CPU usage of a specific process. In this example, we will trace the sched_switch event, which is triggered whenever a task is scheduled on the CPU.
sudo bpftrace -e 'tracepoint:sched:sched_switch { @cpu_usage[args->prev_comm] = count(); } interval:s:5 { print(@cpu_usage); clear(@cpu_usage); }'
This script collects CPU usage statistics for each process and prints the results every 5 seconds.
Output:
@cpu_usage:
"bash": 10
"python3": 7
"chrome": 15
Conclusion
eBPF has revolutionized observability in Linux systems by providing a powerful, flexible, and low-overhead way to gather deep insights from within the kernel. Its ability to dynamically trace system calls, monitor network activity, and profile performance metrics makes it a valuable tool for both developers and system administrators.
With tools like bcc and BPFtrace, users can easily write eBPF programs to monitor various aspects of their system. Whether you’re debugging performance bottlenecks, tracking system calls, or ensuring security compliance, eBPF provides an unparalleled level of visibility and control. Furthermore, the lightweight nature of eBPF makes it ideal for production environments where overhead is a concern.
By adopting eBPF, you can gain deep, actionable insights into your system’s behavior, leading to more stable, secure, and performant applications. As eBPF continues to evolve, we can expect even more sophisticated observability tools to emerge, further pushing the boundaries of what’s possible in system monitoring.