This article can be found at: https://www.ebpf.top/post/ftrace_kernel_dynamic 1. Basic Knowledge 1.1 Default Compilation 1.2 Using the -pg Option 1.3 Using the -pg and -mfentry Options 1.4 Kernel Verification 2. Validation of kprobe tracing mechanism in ftrace 3. Validation using a Kernel Module 3.1 Using kallsyms_lookup_name 3.2 Using Kernel Function Addresses Directly (Pitfalls, Optional) 4. Verification using gdb + qemu Reference In the Geek Time’s “Mastering Container Battles” by Teacher Li Chengyuan, a pondering question was left in the extra session 04 | Understanding ftrace(2): How to Understand the Technology Behind ftrace Tracepoint and kprobe?: Consider this, how can we observe that the first instruction of the corresponding kernel function has been replaced after we register a probe with kprobe for it? Kprobe is a mechanism for dynamic tracking of kernel functions. By using this mechanism, almost all kernel functions can be tracked (excluding those annotated with __kprobes/nokprobe_inline and those marked with NOKPROBE_SYMBOL).……

Continue reading

This article can be found at: https://www.ebpf.top/post/no_space_left_on_devices 1. Understanding “no space left on device” Error 2. Problem Analysis and Localization 2.1 Preliminary Identification of Problematic Function 2.2 Locating the Root Cause of the Issue 2.3 Identifying the Root Cause of the Issue 3. Analysis of Discrepancies Between Code Flow and Tracing Process References Recently, there have been cases of failures in creating containers with the error “no space left on device” in the production environment. However, during the investigation, it was found that disk space and inodes were quite normal. In cases where conventional troubleshooting methods have failed, is there a quick and universal approach to pinpointing the root cause of the problem? This article records the analysis and troubleshooting process using eBPF + Ftrace in a separate environment. Considering the general applicability of this approach, it has been organized in the hope of serving as a stepping stone for further exploration.……

Continue reading